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ABSTRACT 

A project regarding the interpre tabilit y of aata oa 
vocational education outcome data was done (1) to determine iiOk 
different definitions of the vocational student and relatedness of 
training to employment might affect the interpretabilit y of 
vocational education impact data; (2) to synthesize the current 
literature and approaches measuring job satisfaction and job 
performance; (3) to prepare a handbook on vocational education 
tteasures; (4) to prepare position papers on the sociopolitical issues 
underlying the interpre tability issue; and (5) to hold a national 
conference to share the results of the aforementioned activities, ine 
project conclusions include the following: vocational education 
program outcomes need to be closely linked to the values of 
individuals and groups concerned with vocational education; the 
originators of data concerning the outcomes of vocational 
educat ion--the local education agencies-- must be the users of tne 
data in order for ultimate improvements in vocational education to 
result; in general^ placement data are not collected systematically 
with an emphasis on precision and accuracy; and quality of 
instruction^ relevance of program^ impact of program^ and inaividuai 
transition and growth represent four categories of possible criteria 
for which vocational education might be held accountable. (Tnis 
report includes a case study of differences in job placement rates 
reported by states^ and the presentations and reactions at tue 
rational conference, including three commissioned project papers-) 
(EM) 
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FOREWORD 



Interpretinq outcomes of vocational education is a 
major concern of teachers, administrators, advisory councils, 
and policy makers for vocational education at all levels. 
The concerns focus on interpreting outcomes of vocational 
education in response to the requirements and intent of the 
Education Amendments of 1976 and to the need for viable 
vocational education programs and services to prepare 
individuals for employment. 

The National Institute of Education recognized the 
confusion surrounding the interpretation of current voca- 
tional education outcomes data and contracted with the 
National Center to examine the interpre tabi lity of the data 
and examine other outcome measures. This final report of 
the project "Interpretinq Outcome Measures in Vocational 
Education" provides a brief description of the project 
findings . 

Recognition for their efforts is due to many persons 
including the project staff of Floyd McKinney, project 
director; Kenney Gray, research specialist; Marie Abram, 
graduate research associate; and Roseann Pavlick, secretary; 
the evaluation division staff including N. L. McCaslin, 
associate director and former acting project director; 
Jerry Walker, former associate director; Stephen Franchak, 
research specialist and other staff of the National Center 
for Research in Vocational Education. Significant contri- 
butions to the conduct of the project were also made by 
Robert Stump, nroject officer, the National Institute of 
Education; tho national advisory committee for the project; 
Grant Venn, Mary Kievit, and Donald Drewes who prepared 
commission papers; Robert Billings who .wrote a chapter 
of the handbook, ^^ocatj^orval Education Measuresj_ Instruments 
to Survey Former S_t udents and Their Employers ; and Robert 
Morgan, George Copa, and John Jennings who joined the 
previously mentioned individuals in giving presentations 
at the national conference conducted by the project staff. 
The National Center is especially indebted to the many 
local and state vocational educators who participated in 
the case study j)ortion of the project. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
In Vocational Education 
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I . INTRODUCTION 



PROBLEM A\D BACKGROUND 

The overarchino problem for the project was the inter- 
pretability of vocational education outcome data. The 
problem of interprGtina data on the impact of vocational 
education has been persistent and mult if aceted . Congres- 
sional staff members, the National Advisory Council for 
Vocational Education and numerous other agencies have 
expressed concern about the interpretability of vocational 
education outcome data. 

The reasons for difficulties in interpreting vocational 
education outcome data arc numerous. Various reports have 
indicated there are ijirobloms of definitions, problems of 
communication and problems of handling data. Also, there 
is a lack of documentation concerninq the processes used 
to collect placement data. 

Many studies have expressed concern about the virtual 
impossibility of clearly interpreting data on the outcomes 
of vocational education. At the core of this problem is 
the lack of commonly accepted standard definitions of: 
(1) the vocational student; (2) relatedness of training 
to employment; (3) job performance (employ :rs * views of 
former vocational students) ; and (4) job ^ itisfaction 
(former students' views of their occupational and other 
li fe roles ) . 

For "vocational student" and "relatedness," the major 
de f init ional-~and subsequent , interpre tabi li ty--issues 
stem from seemingly inconsistent and incomplete uses of 
the terms as operationally applied by state and local 
education aciencies. The problems with "job performance" 
and "job satisfaction" stem not so much from inconsistent 
usage as from a dearth of measurement tools and processes 
by which one could even l)egin to grasp their operational 
mean inq . 



The project objectives and strateqies evolved from the 
preceding problems. The f>roject objectives were developed 
and the strateqies were designed to detormine the effects 
of the inconsistent and incomplete uses of "vocational 
student" and " re latedness " on i nterpret abi li ty and to 
begin to understand the complexities of measurincr "job 
performance" and "job satisfaction." 



OBJECTJVES 
The project objectives wen*: 

1. To verify inconsistent usage of the terms "voca- 
tional student " and "re latedness " among local 
vocational education a;.;encies and st ate vocational 
e ducat ion agencies . 

2 . To determine how different definitions of "voca- 
tional student" and "relatedness" might affect the 
interpretabi li ty of vocational education impact 
data (e.g., reported percentages of vocational 
students finding related employment after training) 
by using existing data bases. 

3. To synthesize the current literature and approaches 
measuring "job satisfaction" and "job performance." 

4. To prepare a handbook on vocational education 
measures . 

5. To prepare position papers on the sociopolitical 
issues under lying the interpretabi lity issue . 

6. To share and discuss the results of objectives 1, 
2 , 3 , 4 , and 5 by means of a national conference . 

7. To maintain contact with agencies with whom the 
project can contribute and benefit. 
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GENERAL PLAN OF OPERATION 



OVERVIEW 

Several strategies were used to accomplish the multi- 
faceted goals of the project. These strategies included 
the use of a project advisory committee, liaison with other 
agencies and groups, a case study of the factors contributing 
to differences in placement rates as reported by states, 
commissioned papers, a National Conference on Outcome 
Measures for Vocational Education , and the development of 
a handbook. Vocational Education Measures : Instruments 
to Survey Former StudenFs ' and Their Employers ^ A brief 
overview of these strategies is presented in the following 
sections of chapter one. A more detailed explanation of 
the strategies can be found in the remainder of this 
Final Report. 



PROJECT ADVISORY COMMITTEE 

Seven individuals representing a wide range of experience 
served as m.embers of the project advisory committee. The 
committee members were: 

Dr. Ralph Bregman 

National Advisory Council for Vocational Education 

Dr. John Grasso 

West Virginia University 

Dr. Donald Healas 

Cleveland, Ohio Board of Education 

Dr. Arthur Lee 

Northern Arizona University 

Dr. Robert Morgan 

National Center for Educational Statistics 

Dr. N. Alan Sheppard 
Federal Council on the Aging 

Dr. JoAnn Steigor 

Steiger, Fink and Kosocoff, Inc. 

Dr. J. Robert Warmbrod 
The Ohio State University 
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The advisory committee met at the Notional Center on 
November 15-16, 197 7. The committee provided recommenda- 
tions concerning the analysis of extant data, the develop- 
ment of the position papers, selection of authors for the 
position papers, the planning and conducting of the 
conference, and the conducting of the case study. 

In addition to the formal meeting in November 1977, 
the members of the advisory committee served as reviewers 
of the outlines and drafts of the position papers. 
Members of the advisory committee attended the conference 
and served as facilitators for the small groups and reactors 
for the ma^or presentations. Frequent telephone conferences 
were held with committee members having expertise relevant to 
Specific problems. The advisory committee provided a valuable 
link to the potential audience for the project products. 



LIAISON WITH OTHER AGENCIES AND GROUPS 

Throughout the project the staff attempted to remain 
in close contact with other Individuals, agencies and 
organizations whose interests and responsibi li ties had 
implications for this study. Varying degrees of contact, 
through correspondence, telephone conferences, and personal 
contact, were maintained with individuals connected with 
Project Baseline, the National Longitudinal Surveys, the 
National Center for Educational Statistics, the Empirical 
Determination of Nationally Essential Educational Data 
Project, the National Center's Management Information 
Systems for Vocational Education Project, the National 
Occupational Information Coordinating Committee, the 
National Advisory Council for Vocational Education, 
the U. S. Office of Education--Bureau of Occupational and 
Adult Education, and numerous state vocational education 
agencies . 



CASE STUDY 

The overall purpose of the case study was to identify 
and describe the factors associated with the differences 
in job placement rates in field trained and related fields 
as reported by states. 

The issue areas considered in the case study were: 
(1) background, (2) definitions of key terms, (3) process 
of collecting placement data, (4) utilization of placement 
data, and (5) placement function. For each issue area an 
extensive set of questions was developed which further 
clarified and delineated the issue. 



The case study ir.torvicws were conducted in five state 
vocational education aqcncics and in ten local education 
agencies* A complete report of the case study is provided 
in Chapter II and in the A[)pendix. 



COMMISSIONED PAPERS 

Early in the j^roject, three topics were selected to 
be dealt with in tho form of commissioned papers. The topics 
and the b.ors were: 

rspcctivism in Ctioosing and Interpreting Outcome 
sures in Vocational Education/' Mary Kievit 

2. "Oiatcomo Standardization for Compliance or Direction: 
The Criticai Distinction," Donald Drewes 

3. "Criteria Aciainst Which Vocational Education Should 
be Hold Accountable," Grant Venn 

The commissioned pa[.>ers were presented at the Conference 
and are included in Chapter III of this report. 



NATIONAL CONFERENCE 

The National Conference on Outcome Measures for Voca- 
tional Education was held August 16-18, 1978 at the Gait 
House in Louisville, Kentucky. The Conference presentations 
and group reactions are presented in Chapter III of this 
report. A description of the activities undertaken to 
plan and conduc* the Conference can be found in the Appendix, 

VOCATIONAL EDUCATION MEASURES: INSTRUMENTS TO 
SURVEY FORMEI"^ STUDENTS AND THEIR EMPLOYERS 

A handbook of instruments on: (1) job satisfaction, 
(2) job\ per formance , (3) former students' perception of 
vocational tramincj, and (4) employers' perception of 
student's vocational traininq was prepared. The handbook 
provides the practitioner with abstracts of instruments, 
copies of instruments, and a suggested method of choosing 
an instrument that fits the F)ractit ioner ' s purpose. The 
handbook also descr ibes t,he concepts and complexities of 
measuring the four categories of instruments included in 
the handbook . 



The handbook was distributed to the participants in the 
National Conference on Outcome Measures for Vocational 
Education. The handbook may also be available throuqh the 
ERIC system and on a cost recovery basis from the National 
Center for Research in Vocational Education. 



CONCLUS I O NS, HYPOTHESES, 
GUIDELINES, AND RECOMMENDATIONS 



The Interpreting Outcome Measures in Vocational 
Education project addrej^sed a very diverse set of objectives. 
With limited resources it was necessary to focus on several 
issues central to the complex problem of interpreting 
outcome measures in vocational education. 

The audiences interested in outcome measures in 
tional education are also diverse. Complicating the 
to prepare materials for this diverse audience are the 
varying levels of the knowledqe base concerning the in- 
terpretability problem by individuals within each audience. 

As one considers the project conclusions, hypotheses, 
guidelines, and recommendations, it is important to remember 
the contextual features indicated above. 



CONCLUSIONS 

The following conclusions are based on the total scope 
of project activities . 

• Vocational education urogram outcomes need to be 
closely linked to the values of individuals and 
aroups concerned with and interested in vocational 
education . 

• The originators of data concerning the outcomes of 
vocational education, the local education agencies, 
must be the users of the data in order for ultimate 
improvements in vocational education to result . 

• If the data are useful to the provider of data, 
e.g., local administrators and teachers, then 
efforts will be put forth as necessary to collect 
accurate data. 



IB 



• At the time of this study, extant data bases did 
not include sufficient information to permit an 
investigation of the effects of differing definitions 
on placemen t rates . 

• The collection of placement data is viewed primarily 
as a compliance effort by personnel in state and 
local education agencies. 

• In Qeneral, placement data are not collected 
systematically with an emphasis on precision 
and accuracy . 

• Most vocational educators question the reliability 
and validity of r^lacement data. 

• There is a wide ranqr- ^ sophisticat lOr • ; 1 
and data processing ment concerni ^^ir. 

data col lect ion at the- .tate and local leve Is . 

• Quality of instruction , relevance of program, 
impact of procjram, and individual transition and 
growth represent four broad categories of possible 
criteria for which vocational education ' might be 
he Id accountable . 

• The variables which affect the quality of job 
performance are many, making it difficult to 
discern the effects of even very good vocational 
education. However, by (1) measuring dimensions of 
performance which are important for a given job and 

(2) measurincT those variables which affect performance — 
motivation, skills, and role concept--a useful and 
understandable evaluation can be performed. 

• Intrinsic motivation, job involvement, and beliefs 
about work are attitudes deserving increased atten- 
tion by evaluators in vocational education. 

• The effects of vocational education on job satis- 
faction are very complex and difficult to determine. 
By using three models of satisfaction (Lawler's 
component model, job/person fit, and met expecta- 
tions) , the effects of vocational education on job 
satisfaction can be studied and possibly analyzed. 
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HYPOTHESES 

The following research hypotheses were generated as 
a result of the case study findings in the five states. 
These hypotheses represent ideas about what the next research 
steps might be concerning vocational education placement 
data. These hypotheses should form the basis for planning 
additional study. 

1. A state's reported placement rates will decrease 
as the state implements a more systematic place- 
ment data collection process. 

2. A state's reported placement rate will decrease 
as the individuals responsible for collecting 
and reporting the data view the effort more as 
information to be used for program improvement, 
^-r>^•r^■J- hh H ^ compllancc. 

3. As a consistent set of definitions for the key lerms 
regarding placement data is developed there will 

be a decrease in the state's reported placement rate. 

4. A state's reported placement rate will decrease as 
the state develops more specific policies, 

guide lines and instructions , 

5. A state's reported placement rate will decrease 
as the state implements placement data collection 
strategies that use the former student as the 
primary source of in format ion . 

6. A state's reported placement rate will increase 

as the state implements a comprehensive and systematic 
placement system for current and former vocational 
education students . 

7. A state's reported placement rate will decrease as 
the attitude toward collecting placement data 
reflects one of collecting data for program improve- 
ment rather than collecting data to be in compliance 
with federal, state and local regulations. 

8. A state's reported placement rate will increase 

as the percentage of vocational education students 
enrol led in cooperative programs increases . 

9. A state ' 3 reported placement rate will increase 
as the local vocational teachers' commitment for 
placing students increases. 
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GUIDELINES FOR USING PLACEMENT 
DATA AS REI^ORTED BY STATES 



Based on the findinrjs of the case study, the following 
guidelines are suggested for use by those individuals and 
groups using vocational education placement data reported 
by states. These guidelines are general and should not be 
construed to be specific to a given state. They should be 
used as a sot of cautions to observe when using placement 
data as it is current ly being reported . 

1. Users of placement data should be very cautious 
when usina placement data to make comparisons 
within and among states because of the numerous 
ways in wh i c h k o y t e rms are defined. 

Users 1 ac. iin. jit data should be aware of the 

original source c^f the dat i , since there tends to 
bo some difference in the way critical items are 
interpreted by former students in contrast to 
teachers or admin Istrators. 

3. Users of placement data should not automatically 
fault a vocational education program solely on the 
basis of [placement rates. 

4. Users of placement data should be aware of the 
placement services offered Dy the agencies from 
which they are using placement data since agencies 
with more highly developed placement services tend 
to have higher p lacement rates . 

5. Users of placement data should determine whether 
or not the rofjortod data include responses from 
graduates of vocational education programs and 

from non-cjraduate.s of vocational education programs. 

6. Users of the placc.'ment data should be aware that 
in most instances the data have not been verified. 

7 . Use rs of the p laci.'ment data should recognize that 
the job reported for a former student is but a 
step i n the individual * s total career path . 



RECOMMENDATIONS 



As a result of the findings of the project, the follow! 
:oininendations are made . 

• A continuing effort should be undertaken to as- 
certain the values expected of vocational education 
various audiences and how vocational education 
could achieve appropriate linkage witri these values 

• A major research effort should be planned and 
conducted to identify realistic criteria for which 
vocational education should be held accountable. 

• Personnel devolopmont efforts shoi^'^^ itcd 
for prep } .i- i n; dating vocation^ . ^:uucation 
personnel e uncu i ii inci the collection and use of 
placement data. 
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II. CASE STUDY OF DIFFERENCES IN JOB 
PLACEMENT RATES REPORTED BY STATES 



INTRODUCTION 

REQUEST FOR PROPOSAL AND THE PROPOSAL 

The overarch inq problem posed in the National Institute 
of Education's r^-^u:' ;t f<^r nror.^ -nir-.l " ' Mt' ' • i ' 

■ .1 L t oi if ^ \ .> i.iL. i.ntei — 
' . vocaLiop.ui education outcome data. In the 

proposal submitted by the National Center for Research in 
Vocational Education the problem of interpreting vocational 
education outcome data was addressed through the project 
objectives: (1) to verify inconsistent usage of the terms 
"vocational student" and " relatedness" among local vocational 
education aqcricios and state vocational education agencies 
and (2) to determine how different definitions of "vocational 
student" and "relatedness" might affect the interpretabi li ty 
of vocational education impact data. 

In the National Ccntor proposal, the assumption was 
made that access to and/or manipulation of extant data 
bases was not possible to the extent envisioned by the 
RFP. Investigations by the project staff revealed that 
extant data bases that will allow data manipulation to 
determine the impact of varying definitions do not exist. 
NIE concurred with this conclusion. In accordance with the 
provisions specified in the project proposal, the National 
Center project staff proposed a case study methodology as 
an alternative design for responding to the concerns 
relating to the aforementioned project objectives. NIE 
approved the case study alternative. 

The case study methodology was used because it permitted 
an in-depth examination of the Problems related to the 
interpretabi 1 ity of data concerning former vocational 
student pl^^cement in iobs related to their training. A 
distinct advantage provided by the case study approach was 
that it allowed the i r^iterviewer the opportunity to probe 
by asking additional and related questions. The opportunity 
to describe the factors .issociated with the difference in 
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job placement rates in field trained and related fields 
was particularly significant because of the lack of previous 
documentation. There was speculation that certain factors 
may contribute to the difference in placement rates, but 
there was no documentdtion of these factors. It was not 
appropriate to conduct the study by use of a survey instrument. 
The survey instrument methodology assumes that all the 
appropriate questions to be asked can be identified in 
advance of the survey, which could not be done. In addi- 
tion, the survey instrument mothodoloay do. t i - i 
"on the ^oot" 'm-o^ .^r * ,[ qr.l^.\'\:A. <;.,«j at 
" " L ' i nterviews. 

There was little empirical evidence of the critical 
events and activities surrounding the state/local vocational 
education information collection, storage, and utilization 
processes. The case study methodology provided for documenta- 
tion of certain factors associated with the determination of 
placement rates at the state and local levels. 

The point emphasized in the proposal was that the state- 
of-the-art did not now permit hypothesis generation. In- 
deed, if all questions that should be asked could have been 
identified in advance, a substantially different approach 
would have been used for this phase of the project. 



Study Questi ons 

The question addressed by the case study was a descrip- 
tion of the factors associated with the difference in job 
placement rates in field trained and related fields. The 
investigation contrasted the findings among those states with 
high, middle, and low job placement rates. The more 
specific questions addressed were: 

1. What background elements can be ider^tified as 
contributing to the differences in job placement 
rates in field trained and related fields? 

2. How do the definitions of the terms vocational educa- 
tion student, dropout, completer, leaver and 
relatedness contribute to the differences in 

job placement races in field trained and related 
fields? 

3. What processes of collecting placement data can be 
identified as contributing to the differences in 
job placement rates in field trained and related 
fields? 
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4. How docs the operation of a placement service 
contribute to the differences in job placement 
rates in field trained and related fields? 

5. How does tl' Mt-ilLzation of p]:\'' ^v - data c-'^- 

differ' in ]ol) i^lacci -n^ in nr.,i 

ined ..ii ^ it.od fields? 

.ition, soiiio attention was qiven to critical 
soc^ lo-economic and organizational features which may be 
significant in their contribution to the differences in job 
placement rates m field trained and related fields. The 
resources available for this effort do not permit an 
extensive probe of the socio-economic area given its 
enormity. It was tlie intent of this study to deal with those 
issue areas over which vocational educators may have more 
control and/or influence. Specificity of the issue areas is 
provided in the methods and j^rocedures for conducting the 
case study described in the appendix. 



Sample Design 

By using Project Baseline data for 1975, three pools 
of states were determined: those with high job placement 
rates, those with middle level placement, and those with 
low job placement rates. Two states were selected from the 
high and low pools and one state from the middle level was 
chosen in order to examine job placement rates in a compara- 
tive fashion across five sites. Selection of states also 
took into consideration such factors as: (a) region or 
geographic location, (b) urban/rural character of the state, 
(c) state [population ranking, and (d) size of state voca- 
tional education staff. \7ithin each state two local 
education agencies were selected to participate in the 
study . 

The criteria for selecting the local education agencies 
were ; 

1. A larr^e school system and a small or medium school 
system. 

2. The local education agency be located within a one 
hour drive of the state vocational education 
aaency . 

3. The local education agency have a designated director 
of vocational cducat ion . 



The local education acjoncy ha\ 
vocational service areas of fci 



' ast four 
irses to students, 



One local education aqency have exhibited an 
interest in vocational education placement data and 
one have exhibited minimal interest. 



FINDINGS 



In the following presentation of the findings of the 
case study,, states that had high placement rates are 
identified as States A and B. The state that had placement 
rates in the median range is identified as State C. 
States D and E had low placement rates. 



BACKGROUND 

High and Low Placement St ates 

Enrollment . The states with high placement had 
vocational education enrollments equal to approximately 
one-third of their total secondary enrollments. Vocational 
programs were offered in the comprehensive high schools and 
area vocational schools with State B serving a significantly 
larger percentage of its students in area vocational schools 
than State A. 

The enrollments in vocational education were over 40 
percent of the total secondary enrollment in the state with 
low placement. Secondary vocational courses were offered 
only in comprehensive high schools in State D. State E 
offered vocational education in both comprehensive high 
schools and area vocational schools. 

Organizational Posture . Vocational education in the 
states with high and low placement was administered as an 
organizational unit within each state department of educa- 
tion. In the states with high and low placement, responsi- 
bility for conducting student follow-up studies was located 
within a nonprogram unit which performs evaluation and 
related support services for all of vocational education. 
Collecting the student follow-up data was typically a 
one- or two-person responsibility and was three to four 
levels below the state director or head of vocational 
education within each state. In States A, D, and E data 
processing services for tabulating the follow-up data were 
obtained from another division of the state department of 
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education, state r^ov^j rnm^'Ut , (jr an educational institution. 
State B, with relatively lar^;e vocational enrollments, used 
desk calculators to process and analyze follow-up data to 
avoid the time delays and urrors associated with the state's 
data piocessini.; services • 



Mi ddle Placement St a_te 

en rol Imen t . St a t. ^ • C w i. t li the middle rate of p 1 a ceme n t 
had over one^triird of its students enrolled in vocational 
courses. Area vocat. ional scIkxdIs served about one- third 
of the total secondary vocational students and comprehensive 
hi'fh schools SL-rved th^^ otlier two-thirds of the vocational 
enrollment at the s^'condary level. 

Orcfan 1 zat ioiia 1 Posture . Vocational education operated 
under a separate^ state board and was not an organizational 
unit of the state dofjartmcnt of education. Collecting 
follow-up data was the responsibility of an evaluation 
person three levels below the position of the state director 
of vocational education. Vocational program area personnel 
shared in the respon 3 ilj 1 1 i ty to collect follow-up data more 
than vocational i)roaram area r:)ersonnel in the states with 
hicjh and low la cement. 



Fj_rr:ni^ i a 1 Resources 

^^^11 h and Pow 1^1 ace nu • : 1 1 S ^ ate s 

The st.ates v;ith nit;h placement differed considerably 
in the. percentaqe of financial support from local, state, 
and federal sources. The approximate relative amount of 
financial support for State A was 25 percent local, 55 
percent state, and 20 percent federal; compared to 85 
percent local, 6 }.)ercent state and 9 percent federal for 
State B. State A reported ha* ing turned down requests from 
local schools for over 150 vocational education programs 
due to shorta(]e of funds. State B turned away 15,000 students 
(at 45 students per f.)ro(^ram this figure is equivalent to 
over 330 procjrams) duo to lack of funding for local programs. 

Sources of fundina tor State D were about 90 percent 
from local and state^ sourccjs and 10 percent from federal, 
while State E had 62 percent local, 24 percent state and 
14 fKi^rcent federal fundinq of its programs. No information 
was obtained on unmet rociuests for vocational instruction 
[programs for the two states with low placement. 

- 1 - 
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Middle Placement State 



State C's financial support was 83 percent from state 
and local and 17 percent from federal sources. 



Attitudes Toward Evaluation 

High and Low Placement States 

The attitudes toward evaluation across states varied 
from little interest and commitment to great interest and 
commitment- The concern to meet the federal requirements 
for follow-up of former vocational students seemed to be 
the impetus for the evaluation efforts- The less than 
enthusiastic attitude appeared to be fostered in part 
by distrust of the data collected- 



Middle Placement State 

Attitudes toward evaluation expressed by interviewees 
in State C were that the follow-up was necessary to meet 
federal requirements and they questioned the accuracy of 
data collected- 



Cooperative Education 

High and Low Placemen t States 

States reported that students enrolled in cooperative 
education programs often continued in their job on a full- 
time basis after they graduated- The approximate percentage 
of students enrolled in cooperative education was: State 
A, less than 10 percent; State B, 30 percent; State D, 
20 percent; and State E, less than 10 percent- 



Middle Placement State 

Interviewees in State C reported that approximately 
15 percent of the secondary vocational education enrollment 
was engaged in cooperative education programs - 
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Percopt ions Towarci State V oc a t ional F.^ucation Agency 

Hir^h and Low Placement States 

Interviewees were cjueried and observed as to their 
perceptions toward the state vocational education agency 
in each state, in terms of the leadership versus regulatory 
(enforcing state and federal regulations) roles. The states 
with high placement tended to be perceived as "regulatory" 
in their relationship with the local agency and the states 
with low placement tended to be perceived as "leadership" 
oriented in their lunctionincj. This description of per- 
ceptions was used to indicate tendencies or developing 
viewpoints, not firm a'lmi.n i st rative patterns. 

Mj ddl o Plgcem e n t S ta t e 

Perceptions within St^to C were largely a mix of 
"leadership" a:";d " ri>cjulat^ry " functioning. No strong 
tendenc ies suqqes ted a s i nqular viewpoint . 

Unemployme nt Conditions 

High and Low lUacerr.ent S tates 

States A and B with the h.igh placement rates varied in 
their rates of unemployment. State A had unemployment 
typically near the national average while State B's un- 
employment rates exceeded the national unemployment rate 
as well as the national rate for most age, minority, and 
sex subgroups. State R, however, reported that skill 
level labor demand haa remained strong through a period 
of overall high unemployment. 

States C and D, with low rates of placement, had rates 
of unemployment wh ich were ecj ual to national rates and 
tended to follow most overall subgroup changes. 

Middle Placo rTi.:r:t State 

Rates of unemployment m State C were a very close 
reflection of the national rates. 
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Table 1 indicates that all five states offered vocation:Hl 
education in comprehensive high schools but only four states 
offered programs through area vocational schools. State 
D, a state with low placement rates, did not have any area 
vocational schools . 

Vocational education was administered through state 
departments of education in the states with high and low 
placement rates but was administered under a separate 
state board in the middle placement state. 

Financial support tended to be a mix across states with 
a high level of local finances used in a state with a hicrh 
placement rate and a state with a low placement rate. 
Conversely, one state with low placement and one state with 
high placement had low levels of local support. 

Attitudes toward evaluation in all states tended toward 
compliance fulfillment with the two states with low place- 
ment exhibiting a desire to use evaluative findings for 
accomplishing program improvements . 

Cooperative education enrollments were varied. One 
state with high placement and one state with low place- 
ment reported less than 10 percent of student enrollment 
in cooperative programs and one state with high placement 
and one state with low placement reported 20-80 percent of 
students enrolled in cooperative programs. The middle place- 
ment state's enrollment in cooperative education was 
about midway between the levels of the states with high 
and low placement. 

The leadership posture of administrators seemed to be 
stronger in states with low placement. Stater^ with high 
placement reflected more concern with regulation of enforce- 
ment than leadership. 

The factors described in the background on the states 
are summarized in Table 1, "Background Factors by States." 
A discussion of findings on definitions of key terms, 
processes used in collecting follow-up data, placement services, 
and the utilization of data are presented in the following 
sections . 
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Table 1 



Backq round Factors by States 



Factors 



States 



High Place- Middle Place- Low Place- 
ment Rates ment Rates meat Rates 



A 



Types of Schoolij 
Comprehensive 
Area Vocational 
School s 



X 
X 



X 
X 



X 
X 



Vocational Educa t ion 
Admi nistered Throucih 
State Dep a r tme n t 

of Education 
Separate State 
Board for Voca- 
tional Educa t ion 

F Lnancial Support 
Local - 50% 

or more 
State - 50^5 

or more 
Federal - 5"* 

to 20% 

Attitudes Toward 
Eva luation 

Proqram I mprovo- 



».\>opt)rat i Vt^ E<iuc.\i- 
t L on Euro 1 J men t s 

Under 10 5 

10% to 20% 

20% to 30% 



X 



Perception of 

SDVE Role 
Leadership 
Regulation 
Enforcement 



X 



X 

X 



Unemployment Rates 
Nat ion a 1 Average 
Abcve National 
Avorac;<^ 



X 
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DEFINITIONS OF KFY TERMS 



Comparison of Official Definitions 
with Those in Actual Use 

The key terms for which definitions were obtained 
included "student," "completer," "leaver," "dropout," 
"occupational titles used to describe the jobs of former 
students" and "relatedness of employment to training 
received." Official definitions were compared with the 
definitions actually used- An official definition is one 
that was written in an agency's memoranda to state or local 
personnel, documents of policies and procedures, data 
collection instruments and accompanying instructions, annual 
or long range plans, or administrative handbook- Definitions 
actually used may or may not have been the same as official 
definitions- Definitions actually used were obtained by 
observing the various applications of terms and obtaining 
verbalized definitions of terms as used and understood by 
state and local vocational education personnel- Con- 
siderable confusion surrounded the official and actual use 
definitions addressed by the study- 



Student 

High, Middle, and Low Placement States - None of the 
states included in the study had written official defini- 
tions of the term "student" but all operationally defined 
the term as anyone who is enrolled in a federally and/or 
state reimbursed program- There were varying requirements 
that programs had to meet to be reimbursed- Requirements 
such as the hours in a course per day, week and/or a year; 
certificated teacher ; and minimum facilities . There were 
no guidelines at the state level as to when an enrollee is 
a student, i-e-, when preregistration occurs; attending 
class one day, a week, or a month- Local school policies 
operationalized this aspect of the definition by a point 
in time during the first month or whenever official 
enrollments were reported- Differences in time that 
enrollments were reported could have contributed to di f- 
ferences in enrollment and subsequent dropout and comple- 
tion data as students discontinued their training at 
varying times in the first months of the school year. In 
State D there was some reluctance among local education 
agency personnel to label programs or students as vocational. 
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Completer 

High Placement States . The official definition of 
"completer" for States A and B was the federal definition 
of the term. State agencies actually used the definition 
developed by the Committee on Definitions of the National 
Association of State Directors of Vocational Education 
(NASDVE) which is that "a student is a completer who 
pursues occupational training for the duration of the 
course of study or until he/she has gained occupational 
rjroficiency for and obtained employment." The logic of 
this definition as substantiated by state interviewees 
was that a student is a completer when (1) a full course of 
study has been successfully completed regardless of hign 
school graduation or (2) any part of the course or study 
has been completed and the student discontinues his/her 
training and obtains employment. A student was not regarded 
as a completer if enrollment in an occupational program 
was tenninated and employment was not obtained. 

Local education agencies tended to count completers 
as those students who complete all of a course of study 
and graduate from high school. This definition did not 
recognize a student who passes the eleventh grade as a 
completer even though he/she had completed all of a voca- 
tional course of study and was employed. 

Middle Plac ement State . "Completer" was the most 
frequently mentioned term with a definitional problem in 
State C. While the federal definition for completer was 
I)rinted on the state's follow-up form, it v;as unclear to 
most interviewees. Jf the completer was a high school 
graduate, there was no confusion. But there was a large 
problem with employment prior to graduation when all of 
the program offerings had been completed. Briefly stated, 
a person leaving school prior to graduation was a leaver.' 
If he or she obtained employment, the student became a 
completer. Most interviewees used employment in the first 
few months to one year :^fter leaving school as the period 
to determine whether a student v/r.s a leaver or completer. 
One interviewee used this cr:Lterion: any employment i-^' 
the three years following school. 

Low Plac ement States . The official definition of 
a program comj^leter in State D was "a student who finrlsHer. 
a f>lanned sequence of courses, services or activities 
desiqned to meet an occupational objective and which pur- 
ports to teach entry level job skills." Local personnel 
generally interpreted this definition to mean a high school 
tjraduate. 
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state E did not have an official definition of completer 
nor make reference to the federal definition. However, it 
was aenerally accepted that a completer was one who completed 
a planned sequence of training ranging up to two years. 
Some local agency personnel con tended that a vocational 
education completer and high school araduate were tl.o same. 

Leaver 

High Placement States . States A and B differed in the 
definition used for leaver. State A defined leaver as the 
same as dropout and State B defined leaver as a student who 
discontinued (left) a vocational course of study but 
continued as a student of other programs in the school. 
Personnel in local education agencies generally expressed 
unfamiliarity with the term. 

Middle Placement State . There was no official defini- 
tion of leaver in State C and no common definition was 
used. "Leaver" and "dropout" were often used interchange- 
ably. 

Low Placement States . State D officially defined a 
program leaver as "a student who has been enrolled in 
and has attended a program of vocational education (which 
in part is a planned sequence of courses, services or 
activities designed to meet an occupational objective and 
which purports to teach entry-level job skills) and has 
left the program without completing it, except that no 
student shall be counted as a program leaver who is still 
enrolled in another program of vocational education." 
The majority of the interviewees at state and local 
levels, however, indicated that they did not use the term 
or had never heard of it. 

State E did not have an official definition of leaver. 
But the term was commonly used to designate persons who 
(1) left a vocational course to return to comprehensive 
school; (2) left school altogether; or (3) left one 
program to go to another. In brief, it was leaving any 
program without completing it. 



Dropout 

High Placement States . None of the states in the 
study had a written definition of "dropout." States 
A and B used the following definition of dropout: 
"a student who terminates or discontinues school enrollment 
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prior to araduation and does not obtain training related 
employment. " Local education agency personnel used 
this definition of dropout: "any one terminating school 
and not graduating regardless of employment status." 

Middle Placement State . "Dropout" and "leaver" 
were often used interchangeably. 

Low Placement States . In State D, "dropout" was used 
to designate anyone who leaves a program before its 
completion. Some interviewees said it was "leaving before 
graduation." But, for the student who dropped out of one 
vocational program and entered another program, there was 
no indicdtio.: whether the teacher of each program was 
to report the student as a dropout or a leaver. In 
State E, "dropout" was used to describe any person 
terminating before completing the course of study or 
graduating . 



Occupational Tides 

High, Middle, and Low Placement States . None of the 
states in the study had official guide lines as to the 
occupational titles that were used in identifying type of 
employment their former students obtained. State personnel 
reali^'^^d that occupational titles are provided by the 
Dictionary of Occupational Titles and the census; and that 
occupational titles unique to local and area labor markets 
often arise and are used. However, state and local educa- 
tion agency personnel generally used the U. S. Office of 
Education program codes to designate the joL» title of 
former students, even though local education agency per- 
sonnel had a particularly limited knowledge of the U. S. 
Office of Education program codes. In brief, whatever 
a former student called herself /himself on the job was 
the occupational title reported in student produced data. 
In teacher provided data, the occupational title was simply 
what the teacher called the student's work. 

In State D, interviewees reported confusion with 
state and local program cluster designations which were 
different from U. S. Office of Education program codes. 

x^e l atedness 

The re latedness of former students ' employment to 
their training program was not officially defined in the 
five states included in the study. 
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High Placement States . At the state local levels 
in high placement states, actual use of the term "relatedness" 
included any occupation within a U. S. Office of Education 
program code category such as distributive education. 
Frequently a former student ' s occupation v/as counted as 
rel ted to previous vocational training due to commonalities 
of work such as human relations, personal discipline, and 
appropriate work attire. There did not appear to be a 
central theme for iden t i f y ing relatedness . Several 
interviewees indicated that relatedness was a common term 
that had been used for so long that there was no need to 
define it. However, it is important to note that several 
definitions were being used for this key term. 

Middle Placement State . There was no official 
definition of relatedness of employment to training 
identified in State C. The determination of relatedness 
was made by the teacher. Teachers used broad interpreta- 
tions in defining relatedness. 

Low Placement States . State E asked students to 
indicate the relatedness in terms of how much they used 
their vocational training in their current job, using a 
four point scale. The top two points of the scale were 
considered "related" and the bottom two points of the scale 
were "unrelated." However, in States D and E there was 
lack of consensus and considerable confusion concerning 
the definition of relatedness. 



Influences Contributing to 
the Formation of Definitions 

High and ^jOW Placement State s 

The U. S. Office of Education was the principal 
influence in the formation of • de f initions . Generally, 
the written definitions of terms used were those which had 
been developed by the U. S. Office of Education, Bureau of 
Occupational and Adult Education. When a term had not been 
defined by the U. S. Office of Education, states typically 
involved local administrators and teachers in defining 
terms . 
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Middle Placement State 



State C relied heavily on federally derived definitions 
but the state agency was active in attempting to develop 
definitions of terms not defined by the federal agency. 
Effort was being exerted with some local involvement 
to define "dropout" as distinguished from "leaver" and to 
clarify the definition of "completer." 



Federal 

The U. S. Office of Fducation, Bureau of Occupational 
and Adult Education relied primarily on the law and 
legislative committee reports in the development of 
definitions and attempted wide involvement of the vocational 
education professional community. Creating, deleting, or 
combining definitions followed this legislative and profes- 
sional community involvement. 



Strategies Used to Communicate 
Definitions of Key Terms 

High and Low Placement States 

The communication of definitions of key terms was 
accomplished through four strategies. These were: 

1. Instructions which accompany the forms or in- 
struments used to collect follow-up data. 

2. Conferences/workshops held annually in each 
state. States with high placement had one or 
two conferences/workshops annually and states 
with lov^ placement had three or four conferences/ 
workshops annually. Usually a small fraction of 
the conference time was provided for discussing 
definition of terms and other materials used 

in the collection of follow-up data. 

3. Informal personal communication among state and 
local personnel. This strategy was a natural 

part of interaction at the local level among teachers, 
counselors, and administrators, and interaction 
among these local individuals and their state or 
regional agency supervisor or consultant. 
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4. Annual evaluation reports, which included 
certain definitions of terms central to the 
collection and reporting of placement data 
were prepared by States A and D. Evaluation 
reports were not a strategy for communicating 
definitions in States C and D. 



Middle Placement Stare 

Strategies to communicate definitions of key terms in 
State C included: (1) instructions on the follow-up 
forms r (2) an average of three workshops annually where 
some time was given to collection of follow-up data; 
and (3) informal personal communication among state and 
local education agency personnel. 

Federal 

Personnel in the U. S. Office of Education, Bureau 
of Occupational and Adult Education used two strategies 
to communicate definitions of key terms. First, a work- 
shop was held annually in each region involving personnel 
in the Bureau of Occupational and Adult Education, the 
regional office staff, the state director of vocational 
education, and his/her evaluation specialist regarding the 
form used to report state follow-up data including detailed 
definitions and discussion of key terms. 

Second, the staff participated in the May and December 
meetings of state directors of vocational education through 
formal presentations and discussions and informal personal 
contacts to present any new definitions and to clarify 
previously de fined terms as needed . 



Monitoring and Evaluating the Use o f Def initions 

High, Middle, and Low Placement States 

Generally , mon itoring and evaluation of de fin it ions 
currently in use can be characterized as nonsystemat ic 
and uncoordinated. However, in many cases, monitoring and 
evaluation were described as nonexistent except for aata 
processing or another office checking of the data for 
obvious mistakes or inaccuracies. Some interviewees 
questioned the need to monitor definitions due to the 
lack of standard definitions. 
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Federal 



The U. S. Office of Education, Bureau of Occupational 
and Adult Education reviewed the definitions used by 
states in the Management and Evaluation Review for Com- 
pliance (MERC) which is conducted in each state every five 
years. Thus, definitional changes were assessed in ten 
states during the first year following any change to 
determine its usage and success or problems. Further, the 
regional office staff had opportunities annually to review 
state reports to determine the definitions used in each 
state • 



Summa ry 

In the five states included in this study, there was 
a paucity of official definitions to guide the collection 
and interpretation of follow-up data in vocational educa- 
tion. Concurrently, the state and local staffs were using 
definitions which they had created incrementally and are 
struggling to preserve the wording and interpretations 
with which they have had professional involvement and 
identity . 

One school administrator noted, "I have been calling 
this course nonvocat ional for 25 years and I'm not changing 
it now to the vocational classification to serve a new 
reporting purpose." This viewpoint reflects the profes- 
sional rigidity and vested interest in the use of definitions 
and the lack of attention that was given to definitions by 
most persons associated with the collection of placement 
data. 

The only term included in this study that has been 
federally defined was "completer." However, all states m 
the study displayed considerable confusion over the 
application of the definition of "completer*' particularly 
as used by local education agencies when reporting data on 
former students. The middle and low placement states 
we.e more serious in their efforts to clarify the use of 
the term "completer" than states with high placement. 
However, most interviewees did not use nor were they aware 
of the federal definition of the term. 

"Leaver" was perceived as more of a problem and was 
receiving more corrective attention in the states with 
middle and low placement states than in the states with 
high placement, even though it was a problem common to all 
states of the study. 
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"Student," "dropout," "re latedness , " and "occupational 
titles" were not defined in writing and, as a result, 
received different interpretations and uses across the 
states . 

The principal influence in the formation of definitions 
was the rules and regulations to federal legislation and 
the U. S. Office of Education, Bureau of Occupational 
Education - 

The principal strategies used to communicate defini- 
tions of key terms were common to all states and included 
instructions on the data collection forms, annual or quar- 
terly workshops, and personal contacts among state and 
local personnel . 

Monitoring and evaluating the use of definitions was, 
with very few exceptions, nonexistent in the five states 
studied. The process of collecting placement de.ta, the 
function of placement services, and the utilizacion of 
placement data are discussed in the following sections of 
the findings of the study. 



PROCESS OF COLLECTING PLACEMENT DATA 

General Processes Used in Planning 
and Conducting Placement Studies 

High Placemen t States 

The process used to collect placement data in the 
states with high placement rates can be characterized as 
minimal and unsystematic. The collection of placement 
information in both states was viewed as a compliance 
effort. 

In State A, enrollment data were collected in October. 
In early May, the vocational teachers indicated the 
dropout and/or completion status of the students. Six 
months later, a computer listing of completers was mailed 
to the teachers. On the computer listing of former 
students, the teachers provided information regarding 
the placement status of former students. The state voca- 
tional education agency imposed no systematic process 
for collecting the data. The teachers were responsible 
for collecting the placement data. 
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Teachers in Stc te A obtaired placement information 
in various ways. The processes used by teachers to collect 
placement data varied among school systems and among 
teachers within a schoo] system- No evidence was found 
that indicated a systematic mail follow-up system was used 
by teachers to collect placement data. The most frequently 
mentioned source of data was the teacher's "knowledge" 
of the job being held by former students. Teachers in- 
dicated they secured placement information from personal 
visits by the former student, through employers, telephone 
calls to former students' homes, and through friends . 
Some local administrators in State A expressed concern 
about whether teachers were obtaining valid placem.ent 
information . 

In State B, the state vocational education agency sent 
forms to local education agency administrators in October. 
These forms were completed by the teachers or other 
individuals on vocational education graduates in the previous 
years. A state vocational education agency official noted 
that it is a matter of local option as to whether the local 
education agencies complete the form. If the local educa- 
tion agency does not cooperate, there is some threat that 
the state vocational education agency will cease funding 
vocational education programs, but, in reality, this 
never happens. 

The interviewees in the srate vocational education 
agency in State B were unaware of the process used by 
teachers to secure placement data. Officials at the state 
level believed most local education agencies conduct a 
one-year follow-up by use of a postcard. The effort was 
viewed as one of securing compliance information only. 



Middle Placement Srate 

In the middle placement state, the source of placement 
data was the vocational teacher. The process of collecting 
placement data varied somewhat among local education 
agencies, depending mostly on the size of the district. 
In general, the local education agency forwarded the in- 
formation to the state office. Feedback to local educa- 
tion agencies can be characterized as very haphazard. 
The statement was made frequently at the state, regional, 
and local level that there was no organized statewide process 
for collecting f)lacement data. 
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The process used to collect placement data was 
coordinated with the statewide student enrollment system. 
In August a student roster form was distributed to voca- 
tional teachers. The rostc.r (vocational education class 
roster) was distributed to all secondary schools ^ state 
vocational-technical schools, and state universities with 
vocational programs. Early in the school year the form 
was completed by the teachers and returned through the 
regional office to the state office. At the end of the 
school year, the form was returned to the vocational 
teachers and they were asked to indicate the termination 
status of each student. The succeeding January these 
rosters were ma i led to each vocational teacher for use in 
collecting placement in format ion on former students . The 
placement information on former students was recorded as 
of February 15 . 

There was unanimous agreement among state, area, and 
local personnel that there was not a good system for 
collecting placement data. The system in use was com- 
pletely dependent on the teachers supplying the informa- 
tion on former students. Teachers appeared to be very 
individualistic in the way they kept placement records. 
The teachers tended to know more about the better students 
and seemed to ignore the dropouts. Little effort was 
expended to locate former students. If the placement 
status of a former student was not known, that individual 
was marked as "unknown" on the reporting form. Some 
forms had been returned indicating the placement status 
of an entire class was unknown. 



Low Placement States 

There were considerable differences in the processes 
used to collect placement data in the two states with low 
placement. However, it appears the process in both of 
the states with low placement was more systematic than 
that of the states with high placement. 

In State D, the state vocational education agency 
requested all schools to conduct a follow-up of their 
vocational graduates . The state agency supplied follow- 
up forms to the local schools . It was not compulsory for 
the local schools to conduct the follow-up and some schools 
elected not to participate. In the participating local 
schools, the follow-up form was sent to a random sample of 
25 percent of tho former students . Statewide , there was 
about a 25 percent rate of return on the follow-up. 
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In State E, a systematic, thorough and rather closely 
monitored one-year follow-up was conducted. In February, 
follow-up w£:s conducted on all of the graduates of the 
previous June. Leadership for development and continuing 
coordination of the system was provided by the state 
vocational education a ency. State and local education 
agencies placed consi :orable emphasis on securing a high 
response rate on the lollow-up. The percent return on the 
follow-up was 70-80 percent. 

Local education agencies in State E began the process 
of conducting the fc^ low-up in December with the mailing 
of the follow-up instrument. If the instrument was not 
returned in two weeks, a po. card reminder was mailed to 
the graduate. The third contact was made by telephone. 



State Policies Concerning Placement Studies 
High Placement States 

Most state and local personnel in State A indicated 
that state policies for placement studies did not exist. 
A few personnel in State A did believe that there were 
unwritten policies concerning placement studies. Although 
personnel appear not to have been aware of written policies 
concerning placement studies, state law has required that 
the Legislative Council and the Interim Education Committee 
be furnished follow-up placement data on students who 
graduated from vocational education programs. 

State B had no policy concerning placement studies. 
The effort was voluntary and viewed as an activity needed 
to comply with federal regulations. The collection of 
data from public schools was strongly controlled by the 
state education agency. 



Middle Placement State 

The middle placement state does not have an official 
policy concerning the collection of placement data. 



Low Placement Stat es 

In State D, there is no policy concerning placement 
studies. The state vocational education agency encouraged 
follow-up studies but did not exert strong pressure on 
local agencies. 
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In State local education agencies tended not to 
have policy statements about placement studies, but most 
individuals were aware of the state mandate for follow- 
up. State legislation, statements in the State plan for 
Administering Vocational Education , and criteria in the 
State Program Standards of Quality provide the basis of 
policy for follow-up in State" E. A publication of the 
state vocational education agency indicated the follow-up 
is: (1) to provide information helpful to making decisions 
about the improvement of vocational education programs , 
(2) to enable agencies to obtain a more complete and accurate 
indication of what happens to former students, and (3) 
to assist local agencies to measure the effectiveness 
of their programs. 



STATE GUIDELINES FOR PLANNING 
AND CONDUCTING PLACEMENT STUDIES 

High Placement States 

In State A, the local schools were provided with an 
instruction sheet. The instruction sheet contained 
guidelines for local teachers to use in completing the 
state's former student follow-up. Several local school 
personnel were unaware of any guidelines provided by the 
state vocational agency. 

In State B, guidelines for providing information 
about former students were provided the local schools. 
State agency personnel indicated that the guidelines were 
ambiguous and frequently change. In addition, state 
personnel believed little use was made of the guidelines. 
At the local school level in State B it was thought that 
the teachers or counselors who collect the data were not 
providcid guidelines . 



Middle Placement State 

A very fragmented picture exists in regard to guide- 
lines in State C. Interviewees reacted to questions 
concerning guidelines with responses ranging from "there 
are very detailed procedures" to "no guidelines exist." 
In fact, very few guidelines did exist. Most "guidelines" 
that were available were included as a part of the voca- 
tional class roster and these guidelines were concerned 
mostly with the details of properly completing the form. 
It appears there was much latitude within the different 
regions and within local education agencies in each 
region . 
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Low Placement States 



In State D, minimum quidelines were provided, in part, 
because of the limited involvement of local education 
personnel in the collection of placement data. 

In State E, handbooks with specific procedural guide- 
lines for conducting rhe follow-up were provided by the 
state vocational education agency. Local vocational 
education personnel were aware of the handbook and 
appeared to use it extensive ly . 



Assistance Available for Planning 
and Conducting Placement Studies 

High Placement States 

In the states with high placement, no systematic effort 
was underway to provide assistance to those individuals 
responsible for collecting placement data. The state 
vocational education agencies in States A and B provided 
minimal assistance as a minor part of either regularly held 
regional or statewide conferences on vocational education. 
Local personnel in States A and B were unaware of assistance 
provided by the state agency. In State A, local personnel 
indicated that during visits of state supervisory personnel 
there was never any discussion of the placement data 
submitted to the state office. 



Middle Placement State 

There was no systematic and comprehensive program 
in operation to provide assistance to regional or local 
vocational education personnel responsible for planning 
and conducting placement studies in State C. The staff 
in State C had provided assistance in the form of one or 
two hour sessions on an irregular basis for the regional 
personnel designated to coordinate the collection of place- 
ment data. The regional personnel related to local educa- 
tion agency personnel. Their assistance tended to be a 
one-to-one effort. On an irregular basis, local teachers 
received assistance from regional personnel or their local 
education aaency central staff on a one-to-one basis. A 
concurrent session on occupational information was usually 
[.)rovided at the biennial statewide vocational education 
conference . 
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Low Placement States 



In State D , there was very minimal assistance provided 
by the state vocational education agency to local educa- 
tion personnel. In State an individual in the state 
vocational education agency was responsible for coordinating 
the placement study efforts. Assistance for conducting 
placement studies was coordinated through this individual's 
office. The state vocational education agency provided 
annual inservice meetings for area personnel who had 
placement study responsibilities. The area personnel 
provided assistance to local education agency personnel. 



Roles and Responsibilities for 
Conducting Placement Studies 

High Placement States 

In State A, the state vocational education agency had 
the responsibility of developing and providing the format 
for the data needed from the local education agencies. 
Local vocational education teachers were responsible for 
securing the placement data. In the state vocational 
education agency, there seemed to be a realization that 
more standardization, control, and supervision of the 
placement study effort is needed. 

In State B, the responsibility for conducting placement 
studies rested with the local education agencies. The 
state vocational education agency assumed the responsi- 
bility for providing the format for data reporting. 



Middle Placement State 

The role of the state vocational education agency in 
State C was that of provider of forms and compiler of 
data. The local education agencies were charged with 
collecting the placement data. Generally, the data were 
collected by the teacher, although guidance counselors or 
secretaries may have been involved. The local education 
agency forwarded the data to regional offices who in turn 
forwarded it to the state vocational education agency . 
The regional office served as a conduit between the local 
schools and the state . 
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Low Placement States 



In State D , the state vocational education agency was 
responsible for selecting the sample of former students 
to be included in the follow-up and for providing instru- 
ments and mailing labels to the local education agencies. 
The local education agencies were responsible for mailing 
the instruments to former students. 

The development and coordination of the follow-up 
system in State E was the responsibility of the state 
vocational education agency. The state vocational educa- 
tion agency exhibited a strong leadership role concerning 
former student follow-up. Area school personnel were 
responsible for conducting and/or assisting local educa- 
tion agencies in following up their vocational education 
graduates . 



R esource Allocations for Placement Studies 

High Placement States 

In the high placement states, no resources, other than 
those resources needed for instruments and mailing costs 
in State B, were specifically allocated for the collection 
of placement data. The placement data collection re- 
sponsibilities had been assumed by individuals with other 
major responsibilities . 



Middle Placement State 

In State C, staff member.^ of the state vocational 
education agency and area staff had major responsibilities 
for planning and coordinating placement data studies. 
Minimal financial resources were available at the local 
level. 



Low Placement Stat es 

In State D, limited resources, other than resources 
needed for instruments and mailing costs, were specifically 
allocated for the collection of placement data. At the 
state level, it was the major responsibility of a staff 
person to coordinate the collection of placement data. 
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In State E, financial resources were made available 
at the state, area, and local levels for coordinating and 
conducting placement studies. The conduct of placement 
studies could be financed by local education agencies 
from the 5 percent added cost funds they received from the 
state vocational education agency. 



External Influences Affecting Placement Studies 

High, Middle, and Lo w Placement States 

The U. S. Office of Education requirements concerning 
placement data were considered to be the most significant 
influence on the process of collecting placement data. 
In all of the states studied, the local vocational educa- 
tion agencies identified the state vocational education 
agency as the chief external influence affecting the 
placement study process. These external influences were 
not surprising in view of the fact that a majority of those 
interviewed expressed the viewpoint that each level was 
engaged in a compliance effort. 



Monitoring and Evaluating Placement Study Efforts 
High Placement States 

In State A, supervisors in the state vocatic 
education agency reviewed the placement reports submittea 
by the local education agencies. The review was cursory 
and no attempt was made to monitor the process of collecting 
the data or to verify the data. In the local education 
agencies there was no evidence of procedures being used 
to verify the data. No monitoring of the process of 
collecting the data or verification of the data occurred 
in State B. 



Middle Placement State 

There was virtually no monitoring or evaluation of 
the data collecting process in State C. Interviewees 
reported that in some areas it appeared that the placement 
data were "adjusted." 
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Low Placement States 



In State D, there was no monitoring of the process 
of collecting the placement data or verification of the 
data. In State E, there was no verification of the data. 
The process for collecting the data in State E was rather 
specific, allowing the state vocational education agency 
to monitor extensively the collection of the data. 



Problems 

High, Middle, and Low Placement States 

All levels of personnel interviewed were asked to 
state the problems they encountered in the process of data 
collection. The following problems were identified in the 
high, middle, and low placement states. 

1. The quality of the data was questioned in all 
states . No individual interviewed was wi lling 
to say that the data were valid. 

2. There was a lack of information about the non- 
respondents . 

3. Little to no effort was made to secure information 
about the leavers . 

4. The period of time during which placement data 
were collected resuJ.ted in the students not having 
been out of school long enough to be employed or 
to be employed in an occupation in which they 
will remain for a very long period of time. 

5. There was lack of information about the individuals 
who do not graduate. 

6. There were no longitudinal placement data studies 
under way in any of the states. 

In addition to the above problems, in State A the 
following problems were noted. 

1 . State vocational education agency personnel 
generally believed that local teachers do a poor 
job of conducting fol low-up studies . 

2. There was no systematic follow-up process. 
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3. Insufficient funds were available to conduct the 
placement studies . 

4. There was a lack of a clear set of instructions 
for conducting follow-up studies. 

In State B, the following additional problems were 
reported. 

1. A number of interviewees expressed the belief 
that the data were altered at various levels. 

2. Too frequently the reported data were nothing 
better than a guess . 

3. Response from former students tended to be from 
the high achieving students and from the low 
achieving students. 

4. There was lack of a systematic process for following- 
up former students. 

5. The placement data collection effort was viewed 
as a compliance effort. 

6. There was too much reliance on manual data 
collecting and processing. 

In State C, the following additional problem was 
reported . 

1- Local education agency personnel, especially 

teachers, did not see the relevance of the data 
beyond compliance with administrative orders. 

In State D, the following problems were frequentlv 
cited. ^ ^ 

1. An extremely low percent of return on follow-up. 

2. Insufficient time and money to conduct follow-up 
studies . 

No additional problems were reported by the inter- 
viewees in State E. 
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Planned Changes 



High Placement states 

In State A, future plans included : (1) following up 
nongraduates , (2) conducting a follow-up three years after 
e. student leaves school, and (3) conducting an employer 
survey • 

In State B, it was hoped that changes could be made 
that would result in more uniformity of data collection. 
Future plans call for more direction from the state 
vocational education agency. 



Middle Placement State 

A new evaluation program has been pilot tested for 
the past two years and will be operationalized statewide 
in 1979-1980. The process of collecting placement data 
under the new system has been evaluated throughout its 
pilot phase and changes have been made. The new program 
was built upon the philosophy that a region should keep 
useful components of its present system and adopt helpful 
components in the new program. 

The new program will continue to rely on the instructor 
as the source of information concerning former student 
employment. One major change in the program will be that 
responsibilities will be centralized by establishing a 
placement coordinator within each region. Also, the 
developers of the new system have actively worked toward 
standardizing terms through their publication and dissemina- 
tion of an instructor's manual. Further, local vocational 
educators were actively involved in the design and piloting 
of the new system. 



Low Placement States 

State D reports no anticipated changes in the process 
of collecting placement data. In State E, hope was 
expressed that employer surveys and a three -year follow- 
up could be initiated. However, no specific plans have 
been made to implement these changes. 
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Federal 



Most of the individuals interviewed at the federal 
level expressed the viewpoint that the problems of col- 
lecting valid placement data were at the local level. 
There was a feeling that local teachers have limited 
resources to conduct placement data studies and that very 
few teachers were properly trained in this activity. 
Further, it was felt that there was a wide variation 
in the sophistication of the Tjollow-up process used in the 
states. Some states simply do not have enough expertise 
and other resources to do the job. 

Considerable concern was expressed at the federal 
level about the kinds of data being collected.^ Interviewees 
expressed the need for data to reveal student "^objectives 
and the relatedness of vocational education to a student's 
continuing education. There was also a feeling that 
beginning jobs do not always reflect well on students being 
placed in the field for which trained, resulting in a need 
for longitudinal studies. Federal officials expressed 
concern about the validity of the data. The belief was 
expressed that local education personnel did not keep 
auditable records. Federal officials expressed the beliefs 
that until the data can be verified there will be a 
considerable number of guesses in the reported data. 



Summary 

The process used to collect placement data in the 
states with high placement can be characterized as minimal 
and unsystematic. The process used to collect placement 
data in the states with low placement was more systematic 
than that of the states with high placement. However, 
there was considerable difference in the processes used 
to collect placement data in the two states with low 
placement. The process of collecting data in the middle 
placement state was best described as haphazard. The state 
with the lowest placement rate had the most systematic 
and comprehensive process of collecting placement data. 

Considerable variation existed among the states with 
regard to the existence of policies for placement studies. 
One of the states with high placement had a state law 
concerning placement i^tudies; however, most individuals 
were unaware of the law. The other state with high 
placement had no policy concerning placement studies. 
In the states with low placement, one state had no policy 
statement while the state with the lowest placement had 
state legislation and written state policies concerning 
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placement studies- The middle placement state had no 
official policy concerning the collection of placement 
data. 

In the states with high placement, instructional 
guidelines concerning the proper completion of the state 
reporting forms were used. In one state with low place- 
ment and in the middle placement state, few guidelines 
existed, and these dealt basically with instructions 
for completing the state reports. The state with the lowest 
placement rate provided the most specific guidelines for 
conducting placement studies. 

In the states v/ith high placement, no systematic effort 
was under way to provide assistance to those individuals 
responsible for collecting placement data. In one of the 
states with low placement and in the middle placement state 
very minimal assistance was provided. The state with the 
lowest placement rate conducted annual inservice meetings 
and area personnel provided continuing assistance. 

In all states, the state vocational education agency as- 
sumed the responsibility for compiling the federal reports. 
In all cases the collection of the data was an area/ 
region and/or local responsibility. The state with the 
lowest placement rate exhibited the strongest leadership 
role concerning placement studies. 

In all states, certain individuals had responsibilities 
for the placement data collection process. In the state 
with the lowest placement rate, financial resources were 
provided to area and local agencies . 

In all states, the federal legislation and the ac- 
companying U. S. Office of Education regulations rep- 
resented the strongest external influence affecting place- 
ment studies. In the state with the lowest placement rate, 
the state vocational education agency had great influence on 
local and area agencies concerned with collecting placement 
data. 

In the states with high placement, cursory review 
was made of the placement data reports, but no verification 
was made of the data. In one state with low placement and 
in the middle placement state no verification of the data 
or monitoring of the process of collecting the placement 
data occurred. In the state with the lowest placement 
rate, there was no verification of the data; however, there 
was some monitoring. 
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Major problems concerninq the process of collecting 
placement data were: (1) quality of the data; (2) lack 
of information on nonrespondents and leavers; (3) limited 
resources to conduct placement studies; (4) lack of 
systematic and comprehensive collection processes; 
and (5) a compliance mentality. 



PLACEMENT FUNCTION 

Agency Philosophical Position 
Concerning the Placement Function 

High Placement States 

Per Sonne 1 within both state and local vocational 
education agencies of the states with high placement 
agreed that youth needed help in making the transition 
from school to the world of work. There was also general 
agreement that this help should, in part, take the form of 
local school involvement in locating entry level positions 
for students- In State A, the primary responsibility for 
student placement lay with the teacher. In State B, either 
the teacher had the primary responsibility for a placement 
or this responsibility was delegated to counselors with 
the teacher then serving in a helping capacity. 



Middle Placement State 

Interviewees in the middle state (State C) believed 
youth needed help in making the transition from school to 
work and that the local school was responsible for 
supplying placement services to students. In the middle 
state the teacher had primary responsibility for placement 
activities . 



Low Placement St;ates 

Interviewees within both state and local vocational 
educational agencies of the states with low placement held 
the philosophical position that youth needed help in making 
the transition from school to work and that it was the 
responsibi li ty of the local school to supply placement 
services. In State D, the primary responsibility lay with 
the teacher, although other personnel were sometimes 
designated to help in placement activities. In State E, 
responsibility for placement might lay with the teacher, lo 
placement counselor, or regional placement personnel. 



state Coordination for the Placement Function 



High Placement States 

Neither state with high placement rates reported 
high coordination of the placement function by the state 
vocational education agency . In both cases ^ the local 
schools were highly autonomous and the state agency 
served only in an advisory capacity. 



Middle Placement State 

In the middle placement state (State C) , the state 
vocational education agency did not coordinate the placement 
function. The local schools were highly autonomous and 
the state agency served only in an advisory capacity. 

Low Placement St ates 

Coordination of the placement function was high in one 
low placement state and minimal in the other. State D 
reported the schools as being highly autonomous in their 
placement functions with the state vocational education 
agency serving only in an advisory capacity. State E 
reported high coordination of the placement effort through- 
out the state. In about two-thirds of the state the concept 
of area wide placement had been developed. In uhese areas 
the state agency provided coordination and technical 
assistance directly to the area placement office. Where 
no area placement office existed, the state agency provided 
these services directly to the local schools. 



Placement Servi ces Provided 

High Placement Sta tes 

The high placement states provided different types of/ 
placement services for students. State A delegated this 
responsibility to teachers whereas in State B this 
responsibility was delegated either to teachers or placement 
counselors. In both states designated personnel appeared 
to accept this placement responsibility. 

In State A , the teacher had the primary responsibility 
for placing students. This teacher-centered approach to 
placement was consistent throughout State A. Teachers 
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appeared to accept this responsibility and spent the time 
necessary to do the job. Occasionally, help was provided 
to teachers by job placement counselors or by personnel 
in Comprehensive Employment Training Act programs. 
Placement service leadership throughout the state was 
provided by twelve 'job [)lacom(^nt counselors during the 
197!3-76 schuui yuai ; 

In State B, the process of placing students varied 
across schools. In some schools, the primary responsibility 
for placement lay with the teacher^ and in other schools 
the primary responsibility was delegated to a placement 
counselor with the teacher serving in a helping role. In 
either case, the designated personnel appeared to have 
accepted this function. In State B, some interviewees 
reported that because their programs were excellent and 
their communication with business and industry was good , 
their graduates were actively sought by the local employers. 
Under these conditions, placement of their students 
was re lative ly simple . 



Middle Placement State 

In the middle placement state (State C) , the re- 
sponsibility for placement of students was delegated to 
the individual teacher. On occasion, a counselor w?s 
assigned to help with placement activities, but most often 
this help took the form of record keeping rather than 
actual placement of students. Most teachers interviewed 
expressed commitment to placing their students. 



Lo w Placement States 

The low placement states differed in the type of 
placement services provided for students. State D dele- 
gated this responsibility to teachers and State E delegated 
this responsibility to one of a variety of persons: 
teachers , placement counse lors wi thin the school , or area 
wide placement counselors. The low placement states also 
differed in the degree of commitment given by the persons 
designated. In State D, the teachers did not appear to 
spend much time with the placement function whereas in State 
E, much effort appeared to be given to student placement. 

State D did not appear to have a well organized 
placement function. Although the general philosophy 
of the personnel throughout the state placed responsibility 
for placement with the teacher, teachers did not appear to 
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spend much tiii in this activity. In some instances, a 
school secretary relayed employers' requests to students. 
All too oftf^n, little was done to help students find 
relevant work experience while attending school although 
these work experiences were reported as a primary source 
of eventual job placement by interviewees. 

In direct response to low local education agency 
emphasis on placement within the schools in State D, a 
model projecL was recently developed by the state vocational 
education agency. The model is currently being disseminated 
throughout the state in a series of workshop^i. The model 
program hopes to increase the local educational agency's 
awareness of the need for placement of students as well as 
provide a series of steps that will help teachers and 
counselors operationalize the placement function. 

In State E, the process of placing students varied 
greatly from school to school. In some schools, teachers 
had the primary responsibility for placing students. In 
other schools, a placement counselor at the local school 
was assigned primary responsibility with the teacher then 
taking a secondary role. In some regions, a highly for- 
malized area wide placement service was available. Where 
special placement, personnel was available, active job 
development was reported. 

Interviewees in State E estimated that 45 percent 
of vocational students received some placement service. 
They also reported that placement was not limited to 
vocational education students or to recent graduates. 
In addition, they reported that employabi 1 ity skills were 
taught within the classrooms to help students learn the 
skills needed to locate jobs on their own. 



Resource Allocation fo r Placement Servic es 
High Place :nen t States 

In both of the hicjh [placement states no financial 
resources were specifically allocated to support the 
placement function. Forsonnel resources were generally 
available throu.jh teacher and counselor time . However , 
placement duties were secondary responsibilities in that 
these personnel did not receive incentives for their 
services nor penalt ies for lack of service . 



Middle PlacemGnt State 



In the middle placement state (State C) no financial 
resources were specifically allocated to support the 
placement function. Personnel resources were aenerally 
available in the form of teacher time. However, the 
placement task was a secondary res pons ibility of the 
teacher. No incentive was provided for service nor 
was there a penalty for lu'k of service. 



Low Placement Sta tes 

Low placement states dif f ered in the amount of 
resources allocated to the placement function- In State 
D, no financial resources and few personnel resources 
were available whereas in State E both financial and 
per son ne 1 resources were available . 

State D did not allocate financial resources to the 
placement function. Teachers were designated as the 
persons responsible for the placement function^ but no 
tiine appeared to be given to tnem to accomplish this task. 
Teachers did not appear to give much time to this activity. 

State E allocated 5 percent added cost monies for the 
placement and follow-up functions. In some schools this 
money was used to give teachers release time , in other 
schools it was used to hire special placement counselors 
within the school, and in some districts this money was 
pooled to finance an area wide placement office. 



Nature and Extent of Cooperation with Other Agencies 

High Placement States 

One high placement state reported cooperation with 
other agencies in the placement of students in entry level 
jobs whereas the other high placement state reported no 
cooperation with other agencies. 

State A reported good cooperation between some of the 
schools and the state employment --ervices as well as with 
private emplovmeit services in placing vocational education 
students in dtry level jobs. 

State B reported no ties between school placement serv- 
ices and the state employment services or any other agency. 
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So 



Middle Placement State 



Cooperation with other agencies did not seem to exist 
to any great extent in the middle placement state. 
Occasionally an interviewee mentioned cooperation with 
personnel employed under the Comprehensive Employment 
Training Act and the state employment services, but this 
cooperation did not appear to be a general picture of 
what was happenincj throucihout the state. 



Low Placement States 

State D did not have any ties with state employment 
or with other employment agencies for placing vocational 
education students in entry level jobs. 

In State E, the local education agencies sometimes 
did and sometimes did not use the state employment service. 
In some instances, state* employment service employees 
served on local vocational education advisory committees. 
In other districts, cooperative agreements had been negotiated 
with the state employment service personnel to assist the 
local schools with i/lacement activities. In addition/ 
State E reported ^u^mtj .^.>operation with counselors from 
the Comprehensive Employment Training Act Program. 



P roblems 

H i(jh , Midd 1 e , ^ . i rul L ow P 1 a c enie nt States 

The most frequently mentioned problem regarding the 
placement tunction was teacher overload. This problem 
was mentioned in four of the five states (State E being 
the i^xception). In addition to the above problem, inter- 
viewees in State A li ;ted the following problems: 

1. No forr^^al orqan i zat ional structure supporting 
the placement functioi!. 

2. No matr'rials prei^ared by state vocational agency 
that would help teachers with the placement 
function . 

3. Need for an area wide approach to placement. 

4. Low waqes for entrv' level work. 
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Interviewees in State B listf^cl the following additional 
problems : 

1. Lack of teacher understanding of how to do 
placement . 

2 . Lack of state vocational agency leadership . 

3. Reduced quality of training in time-sharing 
schools . 

4. Child labor laws. 

5. Poor economy. 

6. Low wages for entry level work. 

Interviewees in the middle placement state (State C) 
listed the followina additional problems : 

1. Placement is not a specific responsibility of a 
des ignated individual . 

2. No formal linkages to assist with placement. 

Interviewees in State D listed the following additional 
problems : 

1. Former students sometimes take jobs for three or 
four months then leave to look for another posi- 
tion. Thu follow-up survey often catches them 
hictwecn jobs making it appear that the entry 
level placement did not occur. 

2. Sixty to seventy percent of the students go to 
c:ol lege . 

3. SJtudents need to learn employabi lity skills to 
help them look for their own jobs. 

4. Cluster approach to instructional curricula does 
not always qualify student for job without more 
specific training. 

5. Noed to evaluate employer's pei option of quality 
of work. 



-48- 



Interviewees in State E listed the following problems: 

1. Lack of monies for placement function. 

2. Reduced teacher contact with students in time- 
sharing programs . 

3. Students not alwiys i^ware placement help exists. 

4. Labor laws, insurance regulations, and unions 
restrict youth employment. 

5. Competition with personnel in the Comprehensive 
Employment Training Act programs. 

6 . Need to let employers know youth are dependable . 

Low Placement States 

One low placement state reported no cooperation with 
outside agencies in the placement of students in entry 
level jobs whereas the other low placement state reported 
frequent use of the state employment agency in the past and 
was working toward qreater affiliation with this agency 
in the coming year. 

State D did not have any ties with the state employ- 
ment service or with other employment agencies for placing 
vocational education students in entry level jobs. 

In State E, the local educational agency sometimes 
used the state employment service. Interviewees reported 
instances of a state employment service employee serving 
as a member of the local vocational education advisory 
committees. Also, cooperative agreements have been 
negotiated with the state employment service which allow 
full-time state employment service people to assist the 
local schools with placement activities in one area. 
In addition, State E reported cooperation with counselors 
employed under the Comprehensive Employment Training Act. 

Federal 

No pertinent information was collected from federal 
level interviewees regarding the placement function of 
the states. 
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Summary 



All states agreed that youth needed help in the transi- 
tion from school to the world of work and that olacement 
into entry level jobs was one way that vocational education 
should help youth make this transition. How the placement 
function was operat ionali zed differs amona the five states, 
but the differences do not appear to explain the different 
placement categories, i.e., states with hiah placemen:, 
states with middle placement, and states with low placement. 

There did not appear to be any commonality between 
personnel assigned primary placement responsibilities and 
the placement rate of the state. In States A, C, and D, 
the teacher had the primary responsibility for student 
placement. In State B and State E, the personnel primarily 
responsible for placement varied from school to school. 
Counselors, job placement coordinators, and other more 
specialized personnel miaht help teachers assume the 
responsibility for placing students. In State E, an area 
wide concept was being used in most of the state. 

There did not appear to be any commonality between the 
degree of commitment of the designated personnel and the 
placement rates of the states. Designated personnel in 
States A, B, C, and E all appeared to show commitment to 
placement. Designated personnel in State D appeared to 
have minimal commitment. 

No commonality was found between financial resources 
allocated to the placement function and the placement rate 
of the state. In fact, the state with the lowest placement. 
State E, was the only state reportina financial allocations 
for placement purposes. 

Coordination did not differ among the five statos and 
therefore could not be used to explain differences in place^ 
ment rates. In all states the state vocational education 
agency served in a consultative capacity for the placement 
function . 

Cooperati on with outside agencies did not appear to 
explain di f ferences in placement rates . One low and one 
high placement state had cooperative arrangements for place- 
ment with other agencies . The lowest state , State E , had 
the most extensive cooperation with the state employment 
service . 

The most frequently expressed problem was that of 
teacher overload. This problem was reported in all of the 
states except State E. 



•50- 



UTILIZATION OF PLACEMENT DATA 

Compliance or Program Improvement 

High Placement States 

The majority of opinions expressed at both the state 
and local vocational education agency levels indicated 
that the col lection of placement data was primarily a 
compliance effort- A few interviewees at the state level 
saw uses for the placement data beyond that of compliance 
although very little actual use of the placement data was 
reported- No interviewees at the local level appeared 
to see much use for the placement data. 



Middle Placement State 

In the middle placement state, the placement data 
was viewed strictly as a compliance effort- An occasional 
reference at the state vocational education agency level 
indicated that there was awareness that the data could 
be used for decision making- 



Low Placement States 

The majority of opinions solicited at both the state 
and local vocational education agency levels indicated that 
the collection of placement data was primarily a compliance 
effort- A few interviewees at the state level saw uses 
for the placement data beyond that of compliance although 
very little actual use of the placement data was reported- 
No interviewees at the local level appeared to see much 
use for the placement data- 



Dissemination Efforts of the Various States 

High Placement States 

One high placement state prepared a summary report on 
the placement data and one did not. The state that wrote 
the summary report distributed it to state and local 
vocational education personnel- The state that did not 
write a summary report distributed a copy of the U. S. 
Office of Education form only to selected personnel at 
the state level . 



In State A, a summary report was prepared on statewide 
placement data. This summary report was distributed among 
the state vocational education agency staff and sent to 
local school administrators. Reports of distribution 
within the local schools varied. It appeared that some 
schools distributed the information in the summary report 
among their staff and others did not. Neither state voca- 
tional education agency sponsored workshops nor other 
forms of dissemination were reported to exist. 

No summary reporV v/-*s • repared in State B beyond the 
completion of the U. Or f ice of Education form. Copies 
of this form were r. r'.tea to selected members of the 

state vocational e'luc^r ^ j agency staff and to regional 
coordinators- No v;r . t: . v or oral information on statewide 
follow-up data was oi^v-: tr; the local schools. The exec- 
utive secretary for tho -advisory council did not receive 
a copy of the data a.-^. ^ , t.r the U. S- Office of Education. 



Middle Placement Sta te 

No summary report was w^tten in the middle placement 
state (State C) ; however, tne U. S. Office of Education form 
was completed- A copy of the completed form was sent to 
the state advisory council. No other dissemination of 
the placement data was done although the data were re- 
ported to be available upon request. 



Low Placement States 

Both low placement states prepared summary reports 
of the placement data and distributed the reports widely 
throughout both state and local vocational education 
agencies . 

In State D, placement data were collected as part of 
a larger data collection effort. A major report was 
written contrasting the present data with past dat ^ on 
many related dimensions. Graphs, tables, and interpreta- 
tion of trends were part of the major report. The report 
was distributed to the staff of the state vocational 
education agency, to the regional coordinators, and to 
local school administrators- In addition, computer 
printouts were given to regional coordinators for all 
the local schools within their region- In the past, 
the local schools could get a computer printout of their 
own data for the asking- Recently, the state vocational 
education agency has begun charging a modest fee to cover 
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computer costs. Local education agencies visited did not 
appear to ask for the computer printout of their data or 
to share the statewide follow-up report with their staff . 
No state vocational education agency sponsored workshops 
were reported. 

State E prepared a summary report of statewide placement 
data. The summary was widely distributed. Sumn-aries go 
to state vocational education agency staff, local school 
administrators, regional coordinators, local counselors, 
advisory council members, and board of education members. 
In addition to summary reports, the state vocational 
education agency also held surruMer workshops to acquaint 
local school personnel with uses of placement data. 



Uses Made of Placement Dat a 
High Placement State s 

No consistent major uses of the placement data were 
reported in the high placement states. In State A, the 
placement data were reported as occasionally being used 
for program improvement and to account for vocational 
education expenditures. Little or no use appeared to be 
made of placement data for personnel evaluations. 

In State D, the placement data were generally not used 
because the validity and reliability of the data were 
questioned. However, the data were sometimes quoted to 
help recruit new students into vocational education 
programs . 



Mi ddle Placement State 

Interviewees in the middle placement state (State C) 
reported occasional use of placement data for program 
improvement and goal setting. Placement data were reported 
as not being used to evaluate personnel. 



Low Placement State s 

No consistent major uses of the placement data were 
reported in the low placement states. Interviewees in both 
States D and E reported the occasional use of placement data 
in program improvement and goal setting. State E inter- 
viewees also reported the use of placement data to describe 
graduates; help recruit new students; and build cooperation 
with business, industry, and labor. Neither of the low place- 
ment states used the placem.ent data for evaluation personnel. 
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Problems 



High, Middle, and Low Placement States 

All levels of personnel interviewed were asked to state 
the problems they encountered in the dissemination and 
utilization of placement data. The following problems were 
identified in four of the five states. 

1. Underuse of data. 

2. Concern for validity and reliability of the 
placement data. 

In addition to the above problems, State A reported 
the following problems: 

^. Need for longitudinal data. 

2. Need for data interpretation. 

In addition to the above problems, State B reported 
the following problems: 

1. Lack of dissemination of the data to local educa- 
tion agencies. 

2. Difficulty in using placement data for program 
evaluation due to high mobility of students, i.e., 
student's report of job in related field in 
another state does not give meaningful data for 
local program improvement geared to meet the 
local job market. 

In addition to the above problems, the middle state 
(State C) reported the following problem: 

1. Need for interpretation of data. 

In addition to the above problems, state D reported the 
following problems : 

1. Low response rate. 

2. Length of form. 

3. Overuse of placement data as a criterion of 
success in vocational education . 

4. Need for more inservice programs to acquaint 
personnel with use of placement data. 
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In addition to the above problems, State E reported 
the following problem: 

1. Need to collect data that the user felt was important 
to decision making. 



Federal 

Interviewees at the federal level reported the following 
information concerning dissemination and utilization at 
the state level: 

1- Placement data were used for program approval 
only in two states. 

2. Considerable doubt was expressed that placement 
data could ever be used for planning as it would be 
politically dangerous and not responsive to large 
investments in staff and equipment. 

3. Concern was expressed that if placement data 
were used to evaluate programs, the data reported 
to the U. S. Office of Education would quickly 
become inflated and would become relatively useless. 

4 . It was felt that tlie present system requires the 
local education agency to prepare too much paperwork. 
The local education agencies presently see minimal 
use of the information being collected. Reports 
that include interpretation of the data need to 

be disseminated back to the local education agencies. 

5. There was low confidence in the data due to the 
large variet^ of collection procedures among the 
states, nonresponse problems, and other me thodologic?.l 
problems . 

Interviewees at the federal level reported the following 
information concerning the collection and use of the place- 
ment data within the U. S. Office of Education. 

1. There has been a historical change in the dis- 
semination of the federal report on placement 
rates during the 1970s. The trend is toward 
longer and more comprehensive reports. 



Federal reports are disseminated as follows: 
chief state school officer, state director of 
vocational education, state advisory council 
chairperson of vocational education, state advisory 
council executive director of vocational education, 
national advisory council of vocational education, 
state director of community colleges, American 
Vocational Association, and several vocational 
education related magazines- In addition, ten to 
twenty-five copies are sent to each state . 



Summary 

High placement states did not differ from low placement 
states in the dissemination or use of placement data. 
One high and both low placement states wrote and dis- 
tributed summary reports. One high placement state and 
the middle state did not write a summary report and 
had little or no dissemination of the raw data they 
reported to the U. S. Office of Education. None of the 
states made any consistent major use of the placement data 
beyond that of compliance with the federal mandate. 

Two problems were reported by both high and low 
placement states: (1) the underuse of placement data 
and (2) the lack of faith in the data due to concern 
for tfie-:v validity and reliability. No problems were 
common to only high or low placement states. 



CONCLUSIONS 

GENERAL CONCLUSIONS 

Personnel at all levels within all five states viewed 
the collection of placement data as primarily a compliance 
effort. In States A and E, the effort was viewed as com- 
pliance with both federal and state regulations. In three 
states--States B, C, and D — the compliance was viewed as 
reaction to the federal mandate. Interviewees at the 
federal level also believed that the state data were being 
collected primarily as a compliance effort (to meet the 
reporting requirements of the legislation, not program 
improvement, etc. ) . 

It was assumed that collecting data primarily in 
response to a mandate would weaken the incentive of 
personnel involved in the effort. This appeared to be 
the c; se in all of the states. Interviewees at the state 
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vocational education agency believed the placement data 
could be used for general decision making and program 
improvement. It appeared that greater effort could be 
made to use the placement data in all of the states. 

With no consistent internal use of the placement data 
and with compliance serving as the major impetus for its 
collection, placement data were not collected systematically 
with an emphasis on precision and accuracy. State E 
reported monitoring its data collection process. All 
of the other states did minimum monitoring or evaluating 
of their data collection processes. in these states only 
a brief review of the data was made as it was submitted 
to the state vocational education agency. Federal 
interviewees expressed concern over the lack of auditable 
records at the local vocational education level. 

Interviewees in all states and at the federal level 
expressed concern for the reliability and validity of the 
placement data. This concern with the quality of the data 
took the form of questioning its accuracy due to: 



1. 


accidental slippage , 




2. 


purposeful slippage. 




3. 


the lack of systematic procedures. 




4. 


lack of information on students not able 
located, and 


to be 


5. 


collection of the data during the former student's 
first year on the job when job changes are frequent. 



Closely tied to the quality of the data was the wide 
variety of sophistication of personnel and data processing 
equipment among the states. State B reported using hand 
calculators to process their data. The other states used 
data processing equipment , but had varying degrees of 
cooperation with the department in which the data processing 
equipment was housed. The federal interviewees confirmed 
this wide range of sophistication and believed it to be a 
major variable accounting for differences in the accuracy 
of placement data among states. 

In only one state. State E, were specific financial 
resources allocated to the follow-up procedure. The other 
states added the responsibility of the follow-up study to 
personnel assigned other major responsibilities. 
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Lack of finances to hire additional personnel and to 
provide inservice workshops to increase the expertise of 
present personnel appeared to limit the ability of the 
states to provide better quality data. Federal interviewees 
agreed that this lack of adequate funding was a factor 
contributing to poor quality data. 

State vocational education agencies generally functioned 
in an advisory capacity in the collection of placement data. 
State E was the exception in that it had considerable state 
level coordination. None of the states had a written 
policy for the follcw-up process. 

The placement data were disseminated differently in 
the various states. State D collected data beyond that 
required by the U. S. Office of Education. These data 
were reported in a major document that contrasted sub- 
groups of students over the past severa"" years. This 
report was widely disseminated throughout the state to 
local vocational education administrators. State B and 
C did not prepare any report; however, the U- S. Office 
of Education reporting form was completed for both states. 
Distributi' . the U. S. Office of Education completed 

form in tht- • tates was very selective and done on 
a request basis. 

States A and E prepared a summary report which they 
disseminated widely throughout the state and local 
vocational education agencies. States D and E also 
made area and local data ava.^able. State E used work- 
shops as another form of dissemination. 



DIFFERENCES IN HIGH AND LOW PLACEMENT STATES 

The previous discussion concerned the geueral situation 
found in all five of the states. Individual states that 
did not fit within the general picture were specifically 
mentioned. The following four conclusions were based on 
the findings of the study where low placement states 
differed from high placement states. 

1. Low placement states had a higher proportion of 

the secondary school students enrolled in vocational 
education. The low placement states reported 
approximately 40 percent of the secondary population 
enrolled in vocational education whereas approximately 
one^third of the secondary population of the high 
placement states was enrolled in vocational 
education . 




2. The process of collectirxg placement data was 
more systematic in the low placement states when 
compared to the high placement states. 

3. Low placement states collected placement data by 
way of a mail survey direct to the former student 
whereas high placement states used teacher col- 
lected data. 

4. Low placement states had a more narrow definition 
"completers" than high placement states. High 
placement states included students who left the 
program before completion but found work in a 
related field as completers, whereas, low place- 
ment states did not. 

5. Both of the low placement states and the middle 
placement state provided placement data feedback 
to the area and local education agencies. 



The information provided in Table 2 reports tre-^ds in 
consistency among the states. The reader should be aware 
of these trends and not accept them as absolutes or state- 
ments of cause and effect. 



Table 2 



Ccnsistency ot Collecting, Reporting and Using 
Placement Data in the Case Study States* 



DEFINITIONS OF KEY TERM3 

Comparison of Official Definitions with 
Those in Actual Use 

Student 

Conipltiter 

Leaver 

Dropout 

Occupational Titles 
Relatedness 

Influences 0.>.itributing i'onnation 
of Definitions 

Strategies Used to Ominiunicate Definitions 
of Key Terms 

Monitoring and Evaluating the Use of 
Definitions 



TRENDS TOWARD CON SISTENCY 

of Middle Between 

Betwe-^n Between Placement High and 

High Low State with Low 

Placement Placement High Low Placement 

St ates Stat es_ States States States 



Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


No 


Yes 


No 


No 


No 


h 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 



PROCESS OF COLLECTING PLACEMENT DATA 

General Processes Used in Planning and 
Conducting Placement Studies 



State Policies Concerning Placement Studies 



Yes 
No 



No 
No 



No No 
No No 



No h 
{ 

No 



Table 2, Continued 



TRENDS TOWARD CONSISTENCY 



I 

0^ 



Issue Areas 



of Middle Between 

Between Between Placement High and 

High Low State with Low 

Placement Placement High Low Placement 

States States States States States 



State Guidelines for Planning and Conducting 
Placement Studies 


Yes 


No 


No 


No 


No 


Assistance Available for Planning and 
Conducting Placement Studies 


Yes 


No 


Yes 


No 


No 


Roles and Responsibilities for Conducting 
Placement Studies 


Yes 


No 


Yes 


No 


No 


Resource Allocations for Placement Studies 


Yes 


No 


Yes 


No 


No 


External Influences Affecting Placement 
Studies 


Yes 


No 


Yes 


No 


No 


Monitoring and Evaluating Study Efforts 


Yes 


No 


Yes 


No 


No 


Problems 


Yes 


Yes 


Yes 


Yes 


Yes 


Planned Changes 


No 


No 


No 


No 


No 


UTILIZATION OF PliCEMENT DATA 












Compliance of Program Improvement 


Yes 


Yes 


Yes 


Yes 


Yes 


Dissemination Efforts of the Various States 


No 


Yes 


No 


No 


No 


Uses Made of Placement Data 


Yes 


Yes 


Yes 


Yes 


Yes 


Problems 


Yes 


Yes 


Yes 


Yes 


Yes 
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Table 2, Continued 



TRENDS TOWARD CONSISTENCY 









of Middle 


Between 


Issue Areas 


Between 


Between 


Placement 


High and 




High 


Low 


State with 


Low 




Placement 


Placement 


High 


Low 


Placement 




States 


States 


States 


J LaLCg 


States 


PIACEMENT FllNrTTDM 












Agency Philosophical Position Concerning 












the Placement Function 


Yes 


Yes 


Yes 


Yes 


Yes 


State Coordination for the Placement 












ruuction 


Yes 


No 


Yes 


No 


No 


Placement Services Provided 


Yes 


Yes 


Yes 


Yes 


Yes 


Resource Allocation for Placement Services 


Yes 


No 


Yes 


No 


No 


Nature and Extent of Cooperation with 












Other Agencies 


No 


No 


No 


No 


Yes 


Problems 


Yes 


No 


Yes 


No 


No 



*The information provided in Table 2 reports trends in consistency among the states. The reader 
should be aware of these trends and not accept them as absolutes or statements of cause and effect, 



SUGGESTED RESEARCH HYPOTHESES 



The following research hypotheses were generated by 
the project staff in relationship to the study questions. 
These hypotheses were generated as a result of the case 
study findings in the five states and should form the basis 
for planning additional study. These hypotheses represent 
ideas about what the next research steps might be concerning 
vocational education placement data. 

1. A state's reported placement rate will decrease 

as the state implements a more systematic placement 
data collection process. 

2. A state's reported placement rate will decrease 
as the individuals responsible for collecting 
and reporting the data view the effort more as 
information to be used for program improvement, 
rather than compliance reporting. 

3. As a consistent set of defini+-ions of the key 
terms regarding placement data is developed, there 
will be a decrease in the state's reported 
placement rate. 

4. A state's reported placement rate will decrease 
as the state develops more specific policies, 
guidelines, and instructions. 

5. A state's reported placement rate will decrease 
as the state implements placement data collec- 
tion strategies that use the former student as 
the primary source of information. 

6. A state's reported placement rate will increase 
as the state imolevaents a comprehensive and 
systematic placement svstem for current and former 
vocational education students. 

7. A state's reported placement rate will decrease 
as the attitude toward collecting placement data 
reflects one of collecting data for program 
improvement rather than collecting data to be in 
compliance with federal, s'cate and local regula- 
tions. 

8. A state's reported placement rate will increase 

as the percentage of vocational education students, 
enrolled in cooperative programs increases . 
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9. A state's reported placement rate will increase 
as the local vof:^ - ional teachers' cominitment -for 
placing students increases. 



SUGGESTED GUIDELINES FOR USING 
P LACEMENT DATA AS RE P ORTED B Y STATES 

Based on the findings of the case study, the following 
guidelines are sugges+ed for use by those individuals 
and groups using vocational education placement data 
reported by states. These guidelines arc general and 
should not be used as a set of cautions to observe when 
using placement data as it is currently being reported. 

1. Users of placement data should be very cautious 
when using placement data to make comparisons 
within and among states because of the numerous 
ways in which key terms are defined. 



2. Users of placement data should be aware of the 

original source of the data, since there tends to 
be some difference in the way cri tical items are 
interpreted by former students in contrast to 
teachers or administrators . 



3. Users of placement data should not automatically 
fault a vocational education program solely on the 
basis of placement rates. 

4. Users of placement data should be aware of the 
placement services offered by the agencies from 
which they are using placement data since agencies 
with more highly developed placement services tend 
to have higher placement rates . 

5. Users of placement data should determine whether 
or not the reported data include respoiiSes from 
graduates of vocational education programs and 

from non-graduates of vocational education programs. 

6. Users of the placement data should be aware that 
in most instances the data have not been verified. 

7. Users of the placement data should recognize that 
the job reported for a former student is but a 
step in the individual's total career path. 



III. CONFERENCE PRESENTATIONS AND REACTIONS 

The presentations in this chapter were made at the 
National Conference on Outcome Measures, August 16-18, 
1978 in Louisville, Kentucky. Three of the presentations, 
Drewes , Kievit, and Venn, were the result of commissioned 
papers for the project , Interpreting Outcome Measures in 
Vocational Education . 



At the Con f erence , reactions were given to the presenta- 
tions made by Billings, Drewes, Kievit and Venn. These 
reactions are included as a part of the chapter . 

During the Conference, the participants were assigned 
to six groups to interact and formulate reactions to the 
major presentations. A summary report of the small group 
interactions is included in this chapter. 



In addition, the chapter includes the remarks of the 
conference analyst . 



A NATIONAL PROBLEM; INTERPRETING 
OUTCOME MEASURES IN VOCATIONAL EDUCATION * 

I would like to compliment the National Institute 
of Education and thn National Center for Research in 
Vocational Education for the fine work that they have 
done pulling together the various inputs for this National 
Conference. Certainly^ being followed on the program 
by Grant Venn, Don Drewes, Mary Kievit, and Jack Jennings, 
puts one in a strange position, seeing that they possibly 
know :ore and have forgotten more than I will ever know. 

There is kind of a strange "Alice in Wonderland" 
type of situation concerning my job, which is the Director 
of ' ocational Education Data Systems. Everytime one 
ha solid draft of all the issues, the issues change 
or you discover that you have only worked through the first 
or second level of those issues. Certainly, the topics 
that have been identified here are without doubt the most 
critical issues to vocational education. This is the time 
when vocational education is faced with establishing to 
the elected officials and the public at large that voca- 
tional education is in fact producing outcomes that are 
of value to society in proportion to the public investment. 
My current job is to attempt to implement the provisions 
of the 1976 Educational Amendments which called for a 
restructuring of reporting mechanisms and data collecting 
mechanisms in vocational education. As those of you that 
followed the House and Senate hearings and subsequent 
con f erences and legislative processes know. Congress is 
most concerned +-hat we develop measures that are sensible 
and sensitive to vocational education outcomes. 

It is important that we understand the legislative 
history. Legislative history is critical because this is 
where much of the impetus for evaluation has been generatedw 
Congress was so concerned with the status of vocational 
education that in three separate sections to the legisla- 
tion, specific reference was made of outcome measures. 
In Section 161A , authorization was provided to develop 
the National Vocational Education Data Reporting and 
Accounting System (VEDS) and that is the section of the 
law that I am currently responsible for implementing. 



A transcription of a presentation by Robert Morgan. 
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Following these provisions was Section 161B which established 
the NOIC network. This network is concerned with developing 
sound supply and demand information for use in planning 
vocational education programs in the states. 

In Section III, Congress expressed concerns about 
evaluation. In that particular portion of the law, specific 
reference was made to follow-up units and programs which 
purport to provide individuals with job skills. This 
portion of the law also provided for following-up employers 
to determine the extent to which those job skills were 
viewed to be useful by the employer in the work setting. 
In addition, to those concerns in Section 107 and 108, 
and more specifically. Section 108, Congress was very 
concerned with the accountability me ch an isms that were 
being used. 

Congress proscribed a very, very complex planning and 
evaluation process for vocational education. In order 
to understand the legislation, one needs to carefully review 
the House report of which I am sure most of you have copies. 
In the House report. Congress expressed some very important 
ideas. There was concern that inadequate data were being 
collected on the outcomes of vocational education. Once 
again in the House report. Congress prescribed two specific 
measures. One, was the degree of placement in jobs related 
to training and the other was the employer satisfaction with 
the product of vocational education--on the job as an 
employee. Other criteria were identified, such as wages 
and job satisfaction. The exact wording of that language, 
though, indicated that Congress was willing to accept 
further outcome measures as they were developed. 

Our approach in the development of VEDS is to begin 
with the simplest of all questions: What is a vc^r.tional 
skill? Without knowing what a vocational skill is, it 
would be av;fully hard to determine outcomes. Directly 
related to that particular perspective is the whole 
notion of what is a vocational [ rogram, and if we can 
identify a vocational program, a student is more than 
likely to end up beincj an individual who is enrolled in a 
vocational program. The question, then, is, "What is a 
vocational program?" 

We have definitions of vocational programs in two 
[)laces. In Section 191 of the Act, programs are described 
as those imparting job skills or to benefit people in 
acquisition of job skills. In Section 112, programs are 
described as those which [purport to impart job skills. 
When we get down to the or^erational size of the issue, 
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it becomes very diffiov^lt because we must deal with dura- 
tion and intensity. Section 112 requires that we follow-up 
every individual that onters into our program which 
purports to impart job skills. If we look at the historical 
reporting in vocational educat ion , that includes programs 
ranging from the two-hour course to a five-year learning 
experience. And various states have different interpreta- 
tions as to just where that starting point will be. 
Vocational education data systems have been able to generate 
at least some closure on this issue , at least closure to 
the extent that we now have a definition of a vocational 
education student which will leave aside the program. 
The individual becomes a vocational student at the point 
of entry in the first unique course or a sequence of 
courses that will lead directly to a job skill. We have 
separated that particular type of student from those in 
the more general vocational program. In structuring the 
forms this separation has been continuous. 

We have also separated out short-term adults. An 
adult under the current law is basically a person who is 
neither a secondary student nor a postsecondary student 
nor one pursuing an AA degree. That is quite a residual. 
It ranges from the two hour course that I talked about to 
a five-year program. In this particular situation, we have 
dichotomized people into long term and short term programs. 
We have not settled the notion of what is a vocational 
student, but have resolved more, what is not a vocational 
student. A very significant question that still remains to 
be answered is: "At what point in terms of intensity, do 
we consider an individual to be a vocational student in the 
program that purports to impart job skills?" At any rate, 
we do have some operational definitions and that probably 
is a step forward. How good they are and how meaningful 
they are are still subject to research and question. 

In terms of our separation of outcomes, we really have 
two ways of looking at outcome in the current data system. 
The first is a very simple kind of measure. How many people 
have completed? Over a period of time now Congress has 
become more concerned, not just with how many people have 
completed, but with what types of people have completed. 
What target groups have completed? Historically we have 
had a process oriented reporting system. Congress has 
moved that emphasis to the output side. And categorically 
so, by race, by sex, by handicapping conditions, ord by 
disadvantaged. The question remains, how many people have 
completed? We do not have data on that at the present 
time, but by 1979 we will have. 
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The more important criterion is secured after the 
individual leaves school. Does the individual find (1) 
employment, (2) employment in job related training^ and 
(3) does the employer find that individual's job preparation 
satisfactory. We have taken an approach of a very minimal 
data system for national purposes in this area. Basically, 
what we are looking for are those completers and leavers 
from programs that purport to impart job skills. We are 
following up a sample of completers and a sample of leavers. 
What we are asking is very simple: (1) "Ar^^s you employed?" 
and (2) "Are you employed in the field relatea to training?" 

Relatedness is to be determined in three ways. The 
best and simplest way is to ask the foii'mer student. A 
second way is for professional judgment to be exercised 
because of the degree of relatedness comparing the cur- 
riculum with the job being reported by the student. And 
a third way which is not completely developed is to match 
occupational classifications to vocational education 
program codes . This particular technique over time is the 
preferable technique since this is replicable, empirical, 
it has a set of rules attached to it, and we will get 
readily comparable data across the nation. Our problem 
at the present time, is that the technology has not caught 
up with the need for information. So in the interim, we 
will be utilizing three approaches. The data are going to 
be less hard than we would like and I would hope that one 
r f the outcomes of this conference would be to discuss 
these mechanisms as well as others for determining related- 
ness and come to some recommendations about how we can 
better pursue what I am telling you is the current status, 
and not what wo would like to do. 

Relatedness of the job is the single most important 
value of the criterion. Did the individual get a job in 
a related area? And what strikes me as very strange, is 
that every person you talk to about relatedness comes up 
with a different reason about why it is impossible to 
determine relatedness. In one instance where an auto 
mechanics program was followed-up, it was learned no 
individuals were qoinq into auto mechanics, but everyone 
was being employed. Further investigation revealed these 
individuals were being employed as heavy machine set-up 
men which did not fall in the teachers' evaluation of a 
related job skill. However, the individuals were making 
about two dollars an hour more than the average autc mechanic. 
So, this issue has been discussed on and on, don't let me 
give you the impression that nobody has thought about it. 
However, it has not been resolved. 
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The second criterioi. that we are more concerned with is 
the empirical basis in terms of relatedness. On the one 
hand, we are taking the inaividual's programs, setting up 
the matrix, and then arraying this matrix of programs 
juxtaposed with column headings of two-digit SACs to 
determine empirically where individuals who receive 
specific training end up in terms of gross population 
categories. At [jresent, I know of no data that really 
reflects this level of det.iil and we believe that this 
will begin to give us the empirical base that we rr^ed to 
make our decisions. 

In almost every state, there currently exist two kinds 
of vocational programs. There are what are known as state 
plan programs and what are called non-state plan programs. 
In some states these programs are classed as vocational 
education and in • -.Trie states they are not classed as 
vocational educa^. ^ ^ The most serious problem is measuring 
the outcome. One ].. jhlem that really needs to be addressed 
is: "How do we get a handle on the non-state plan data?" 
In the government structures of most states, there is 
in-^uf f icient freedom to get data from all sectors which 
oflcr programs with titles similar to chose in vocational 
educration as reported to the federal government. If 
v^e . I ^ ever going to get a handle on the true outcomes of 
vorutic>tnai education, we must at a minimum have a notion 
of ..t is going on outside the state plan. Just to show 
yoa the severity of the problem, I'll relate this classic 
example to you. I had an individual call just a few days 
ago, and for the first time we said report only state plan 
data. Most states don't have any other information in their 
system. Many don ' wan*-, to have any other information in 
their system, but state plan data. However, in zhiu state 
it was going to cut the enrollments in half. In other 
cases every student that is in vocational education, 
whether or not the program is covered by the state plan or 
not, is in the reporting system. When we are talking about 
outconaes, the vocational education data system will only be 
getting those kinds of outcomes that are associated with 
programs covered by the state plan. I think that is the 
most signi f icant issue when we look at our problem and 
the fact that job relatedness really constitutes the two 
issues that are causing the most problems in implementing 
the vocational educa ^o-' measures . 
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P ERSPECT IVISM IN CfiuOuiNG /\ND INTERPRETING 
OUTCOME MEASURES IN VOCATIONAL EDUCATION* 



Purpose and Comp lexi t I es 

Durinq the tjer.'od of the late sixties and into the 
seventies , some writers have captured oublic attention 
and turned that attention to some of the negative outcomes 
of salient social and individual values. Future Shock , 
Z_Gro G rowth , by the Club of Pome come immediately to"lT\ind, 
as does a more recent volume Smal 1 i_s Beautiful . These 
books, among others, are manifestations of reexaminations 
of social directions, social values, and the implications 
of these for the present and the future . These reexamina- 
tions usually include some "pointing of fingers," attributing 
blame, and education at all levels has had its portion of 
these. There is a shared concern in z^any sectors for 
answering the questi on of "How does what we produce , 
provide , accomplish , match the needs, wants, expectations 
of our const ituencies ?" In the educational arena the data 
to provide highly reliable and valid answers are sparse. 
The wisest answer from many vantage points is to focus on 
processes for seekina the kind of data which will provide 
these answers. Yet it is, nonetheless, valuable to ask, 
"What do we know?" and "What can we reasonably in'f'er from 
what we know?" V'hat are the implications for action? 

The three-fold purpose of this paper is, to explicate 
as much as existina knowledae enables us to, the values a" 
the var lous groups d irect ly concerned with vocational 
education; to analyze the impact that these values have 
on the perspective thruuah which members of each group view 
vocational education; and to ascertain the implications 
for choosina and interpreting outcome measures in vocational 
e ducat ion , 



Presentar ion by Mary Bach Kievit. 



The task as delimited sounds relatively simple and 
straight forward. Jn reality it did not prove to be so. 
Delving into Sii. streams of literature showed that tL,.re 

are many definitions of v-olucs. The information available 
on values is very uneven in terms of the groups from which 
the data have been derived; the time period at which it 
has been obtained; and the direct relevancy to vocational 
education of the definitions of values and hence, the data 
collected . 

Some other concerns emanated from the mult idimensionality 
of groups and individuals . I-^ach person holds values for 
her/himself as an individual, and holds values relative 
to the many roles each performs, e.g., as educator, but also 
as parent; as legislator concerned about the social welfare, 
but also as legislator concerned about reelection. In 
brief, there is a multidimensionality about each person 
and about the concerns held by various croups that make 
definite statements hazardous, as guidelines for action 
in this area, and compel us toward tentative statements to 
be tested for soundness in the diverse situations to which 
a concern for values has import. 

In an era touted for the rapidity of change, the question 
of stability and change must be confronted directly both 
in the lonqer historical context and also within the lifetime 
of individuals, V^hile considering the question of stability 
or change, the nature of change and the sources of change 
come into play. Change can be said to occur when one moves 
from the diffuse, the aeneral, to the differentiated, the 
specific. But change of this nature does not preclude 
changinq the degree of importance accorded one value as 
it comes in conflict with another; or indeed of ceasing 
to value what had been valued. 

Education oresuirably has some part to play in change. 
(Opinions as to what that oart is and should be varies among 
educators. Some hold that education should inculcate 
specific values. Others hold that education provides the 
experiences and the substance from which individuals form 
values, in a highly individual way, so that education 
inf luences value formation on ly indirectly . Sti 11 others 
contend that the or ope r role of education is to directly 
facilitate the clarification of values^ and some rational 
examination of the implications of holding specific values. 
As one seeks tentative answers to these questions/ one must 
make decisions about the nature of evidence which one will 
employ . For the literature provides evidence emanating 
from a deductive mode, the introspective, logical, analytical, 
philosophical, and the behavioral science. 
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In addition, accofjtance of a specific definition of 
value, and clusters of values, enables one to use data as 
indicators of the presence or absenr^ of valuing and in 
so doing to draw inferences. 

In each case , the do 1 i nc,*aL ion of the reality investigated, 
both through the definitions, and the subsequent operational! za- 
tion into systematic measures, creates inherent limitations. 
Self-report measures, for oxam[>le, raise the serious question 
ds to the level of awareness of individuals regarding their 
values, the extent to which f^rede termined labels have, 
in fact, distorted the reality of values for specific 
individuals and specific a roups . 

Having shared with you some sense of issues to be 
resolved in the process of developing a position, let me 
now share with you the results of that process. 



Conc eptua j IYj.mework and Types of Evidence 

The conceptual framework within which I chose to 
inquire is that of the behavioral sciences. Sociologist 
Robin Williams is rather cenerally recognized ar one of the 
more profound thinkers among sociologists and has provided 
a thoughtful analysis of value orientations which exist 
in the ^^anerican culture. Social-psychologist Milton Rokeach 
has built upon his own inquiry in beliefs and attitudes to 
study further the nature of values held by individuals, 
and the way in which individual values differ among persons 
based on inclusion in socioeconomic categories, occupational 
categories, racial and religious categories among others. 
Rokeach is knowledgeable of Williams' work and draws upon 
and expands some aspects of it. Another approach closely 
linked is that of Abraham Maslow, the humanist psychologist 
who postulated a hierarchy of basic sets of human need. 

Relative to the types of evidence, greatest weight has 
been given to knowledge derived from empirical data through 
systematic research. Due to the limits of this source, 
other types of data have been utilized as a basis for drawing 
inferences. Some of these are in the category of informed 
opinion. Some are derived from testimony to legislative 
commitLees. Some are quite impressionistic based on 
personal experiences with parents and students throuc;!! 
time. In brief, as a result of the significance of the 
subject, I ha-.-e employed all of the information I had. 
Thus, in Spino-'^ian terms,, it ranges from hearsay, through 
vague experience, knowledge reached by reasoning, and by 
immediate deduction and direct ^:erc^.ption , with an effort 
to use the las t two as much as ; - .^s ^ible . 
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Values; Khat Are Th ey? How Do Wc Discern Them? 



Values, according/ to Williams, is "any aspect of a 
situation, event, or object that is invested with a 
preferential interest as oeing 'good,' 'bad,' 'desirable' 
and the like." (Williams, 1956, p. 374) Values have a 
conceptual element; they are affectively charged representing 
actual or potential emotional mobilization; they are 
not concrete goals of action but criteria by which r;oals 
are chosen; they are important, not trivial or of slight 
concern. (Williams , p. 374) 

Socia l values are regarded as matters of collection 
welfare by an effective consensus of the group. In sum, 
both for groups and individuals "values are modes of 
organizing conduct — meaningful, affectively invested 
pattern principles that guide human action." (Williams, 
p. 375) 

Values are viewed as constituting a type of continuum. 
Values concern the goals c^f action in addition to being 
conponents in the selection of means to achieve the goals. 

Empirically some of the evidences of values are: 
the choices made; the directions of interest (that to 
which people pay attention); statements ci what is valued, 
emotional reactions to statements, implicit premises — 
not stated because they are taken for granted, explana- 
tions and reasons given for conduct; rewards and punish- 
ments, i.e., social sanctions. (Williams, pp. 373-382) 

Williams differentiates between dominant and subordinate 
values noting that for a group or system as a whole these 
crite\ia are useful: 

• extensiveness--proportion of population and 
activity manifesting th^ value, 

• the duration of the value — has it persisted 
through time, 

• the intensity with which the value is sought or 
maintained--e. g. , promptness, certainty, and 
severity of sanctions when the value is threatened, 

and 

• the prestige of value carriers, persons, objects or 
organizations considered to be hearers of the value, 
(pp. 382-383) 
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In a pluralistic society such as this, precise and 
detailed characterization of values can only be done for 
carefully delimited seqments of society. In view of the 
need, however, for some core values for any society to have 
the integration essential to survival, major value orienta- 
tions are apparent. Different croups within society may 
place the value components of such major orientations in 
different positions of dominance and subordination and may 
omit some, and add others. 



Ma] or V alue c^rien t a t i on s 

Utilizing the above criteria, V^illiams, in 1956 
examined available evidence to specify major value orienta- 
tions in ^^merican society. The validity of his analysis 
IS attested to by the commonalities found with analyses 
completed in the 1970 's to address the question of stability 
and/or chanqe in values in America. These r^ajor value 
oi entations have import for this inquiry because the values 
of the various cjroups concerned with vocational education 
emanate from 'his social-cultural base. 

Major value-orientation, as ideal types and kinds of 
central tendencies by v/hich to note departures, according 
to Williams are: 

1 . Achievement -success : stresses personal achievement 
especially secular occupational achxC-vement , 

and acquisition of aenerally accepted symbols 
of success. 

2. Activity and work: mastery and domination of the 
external world with directed and disciplined activity 
m a requl^r occupation a particular form of the 

bas ic or ieri tat ion . 

3- Moral orientation: an ethical quality of a par- 
ticular ^jype that includes principles to work Iiard, 
lead an orderly life, to have a name for integrity 
and fa\r dealinq, not to spend one's substance in 
rt.ckless display, to have the resolution to carry 
out ^he purposes you undertake . 

4. Humc^nitarian mores: disinterested concern and 
helpfulness, including personal kindliness, aid 
and comfort, and organized philanthropy. 



Eff ciency and practicality: unites activity 
and substantive rationality , f ocus ing upon a 
choice of the most effective means for a given end. 
Manifestation is the appreciation of technical values 
in skilled trades , technical , quasi-professional 
and professional vocations with bystematic in- 
doctrination in the standards of 'doing a good 
job." "Practicality as to concrete goals of 
action correspondingly has meant the canalizing 
of ac' on in the service of those specific life- 
inodel most highly approved in the general culture — 
broadly speaking, rational, strenuous, competitive 
striving for personal validation through occupa- 
tional success" (p. 403). 

Progress: emphasis on the future rather than the 
present or past, receptivity to change, faith in 
the perfectibility of the common man--optimism. 

Material comfort : high level of material comfort 
is sou^jht after; standard of living has its attached 
meanings however for symbols of success, competence, 
and power. 

Equality: a value complex subject to much strain 
includes equality in interpersonal relations as 
a goal and s tandard--wi th compromises in practice; 
equality of specific formal rights as in the strong 
and continuing strain for equality for legal rights 
ror r^\l citizens; equality of opportunity to 
cconomi : f reedorn and individual achievement 
raL'ior than equality of condition in either 
economics or achievement . 

Freedom: for the i.ndividual as an integral agent 
re latively autonomous and morally responsible . 
The corollary is that a great variety of forms of 
personal dependencies constitute a loss of freedom 

or example, "freedom of thought so that the 
truth may prevail; freedom of occupation so that 
careers may be open to talent ; freedom of sel f - 
government, so that no one may be compelled against 
his will" (Becker--Williams , p. 434). 

External conformity: in a very heterogeneous culture 
conformity in externals helos to make it possible 
to continue the society in spite of many classes 
of interests and basic values. 
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11. Science and secular rationality: an emphasis 
reflecting the rationalistic--: ndi vidua lis tic 
tradition; i.e., discip],ined , rational, functional, 
active, requiring systematic diligence and honesty. 
It is compatible with strivings for mastery of 

the environment, denials of frustracions and 
refusals to' accept the idea of a f i.;ndamentally 
unreasonable and capricious world 

12 . Nationalism — patriotism: widespread satisfaction 
of people with the country. 

13. Democracy: based on the implicit belief in natural 
law as opposed to personal rule and in the moral 
autonomy of the indi^^idual with a theme of democ- 
racy as a procedure xn distributing power and 

sett I ing conflicts . 

14. Individual person-il i ty : the valuing of the develop- 
ment of ind ividual personality to th end that the 
person is independent, responsible, and self- 
respectinc and thus worthy of concern and respect 

in one ' s own right; in sum, valuing a certain kind 
of individual. 

15. Racism and aroup-superior ity : the ascription of 
value and privilege to individuals on the basis 
of race or particularistic group membership 
according to birth in a particular ethnic group, 
social class or related social category. A per- 
vasive and powerful counter-current to the values 
of equality, human i tar i an values , political 
freedoms . 



Ch ange or Stability ? 

Williams formulated these major value orientations 
in 1956. Have- these persisted? Have these changed? 
Alex Inkeles (C han ge , 1977, p. 25) has examined American 
[:>erceptions to look for continuities and discontinuities 
with the past. A comparison of the areas he treats as 
"perceptions" with the major value orientations of Williams 
shows a number of commonalities. Inkeles states "... 
the national profile is still consistent and often 
contrasts sharply with that of other nations, according 
to substantial psychological test results and public 
opinion aata." The accumulating evidence is unmistakable: 
"Over a span of at least 200 years there has been a marked, 
indeed a remarkable degree of continuity in the American 



national character. But the evidence tells us too that 
certain prominent changes are also occurring . . •• 
(p. 26). 



One of the continuities is the continued intensely 
held belief in the special qualities of the American 
systein--a special brand of patriotism. with 12 percent 
preferring to live in some other country (lower than 
for nine Euroo-r.n countries) the large majority were 
committed to this country even in the light of a sharp 
erosion of confidence in basic institutions, includina 
government. ^ 

Americans still believe in the power of an individual 
to shape his or her future. This belief is held not only 
by professionals and business people but regularly by 
two-thirds to three-fourths of American blue collar 
workers. The emphasis on self-reliance and independence 
continues . 

The majority of Americans believe that most people 
car be trusted; that we can transform the physical and social 
wor,;c. and even human nature over to our own satisfaction. 
The T.Tjority have optimism in their personal economic 
futi'.- :s; an openness to new .experience and innovation 
extending beyond the technical and mechanical to the social 
and sensate; a continuing prope..sity to assert their rights 
of personal autonomy over public control. Inkeles con- 
siders the evidence and concludes that also continuing are: 

a sense of intrinsic worth and a conviction that one is 
equal to all others before the law; individualism, restless 
energy; pragmatism; orashness or boastf ulness ; this 
wordline^o; a preference for the concrete; and a certain 
discomfort in coping with aesthetic and emotional expres- 
sion" (p. 29, . 

Changes according to Inkeles include: "a substantial 
and steady increase in tolerance of religious, sexual, 
and racial differences; increase in inner-direction 
vs. other direction; a shift from the number of people 
who considered t^c most important attribute of a job to be 
Its intrinsic im^ .rtance or its promise of advancement to 
high income and shorter hours as being more attractive. 
(Evidence of valuing the ability to purchase symbols of 
success, achieve mc.terial comfort and have gre jiter freedom, 
through more discretionary time?) Seniority and experience 
were rated as the basis for gettinc- ahead in their line of 
work by more persons than those mentioning hard work and 
persistence. The consumption ethic se-ms to have re- 
placed the Portestant ethic of -aving and Investment, 



- /»- 



ERIC 



as evidenced by the use of rrredit. Although he notes 
what may be some strain and conflict through some seeming 
inconsistt?ncies , he believes that- the chanaes do not 
undermine the fouidation.s of t:i. system (p. 32). 

Vocational-Technical Kducation as a Means of 
A chieving Goals Consonant with 
Social and I ndivid ual Values 

Vocat ional Educ ato rs, r.e gislators , Employers 



Education , aenera 1 1 y , and vocational- technical 
I'ducation, specifically, have been and continue to be 
valued n:;eans of achieving goals directly related to 
social values which take form in individual values. 
Its continued support is linked to its credibility as an 

t f i c ien t and practical means to achieving those social 
and individua l ends. That credibility is not exclusively 
based on hard data from evaluative research, but, presently, 
of equal or greater significance is, the belief of a 
sufficient number of vocational-technical educators, 
employers, parents, and students that it contributes 
significantly to social and individual goals. Further- 
more, the valued social and individual goals and the goals 
to which vocational-technical education is a means have 
remained fairly stable over the past fifty years, in the 
most general sense. Since 1963 however, vocational- 
technical education has been charged quite explicitly to 
address itself to extending its service to more categories 
of persons vxewed as having the right to and need for 
vocational education to further the social values of 
equality of opportunity through accessibility. 

Ainong the ack now:. edged leaders of vocational educators 
and leaders of the constituencies served are Charles 
Prosser, Terrel Bell, James Rhodes, elected official 
(businessman and legislator), John W. Thiele, and Roman 
Pucinski of the National Advisory Council on Vocational 
Education, and v;illiam Pierce, former Deputy Commissioner 
for O'rcupational and Adult Education- 

In their work Vo cat ional Education in a Democracy, 
Prosser and Allen at tribute the development of vocational 
education to the valuing of efficiency applied to mother 
value, i.e., training and acquisition of occupational 
"intelligence" which exceeded that present in the old 
"pick up method" or haphazard vocational training. 
Furthermore, education generally is viewed as the means 
for securing stability and progress in a democracy, 
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with vocational education a special form in its relation 
to the development of the material and human assets of 
a peorie (p. 19). [Mastery over the external world.] 

Vocational education conserves natural resources and 
human resources. [Efficiency and practicality.] The 
first by promoting, disseminating and transmitting skill, 
knowledge and the results of invention and by conserving 
human effort. The second it achieves by promoting morale 
and intelligence by workers (p. 19). Indee i Prosser and 
Allen assert that no form of vocational education worthy 
of the name would confine itself to manual skill and general 
technical knowledge. Vocational education must also give 
that "special thinking intelligence, 'I' which functions 
in the given occupation" (p. 44) . "This kind or use of 
man's job intelligence is shown whenever a master of any 
occupation brings to bear all his knowledge to think 
his way through some difficulty that must be overcome." 
"... Merely organizing occupational experiences for 
training as a substitute for the old pick up method is 
some improvement. But it does not get us very far unless 
both processes and functioning facts are so taught that 
they give understanding to the worker and habits of re- 
sourceful thinking with these facts in the processes, 
situations, and opportunities of his employment. Only in 
this way can the native ability of any people be utilized 
to the full in the economic field" (pp. 44-45). 

James A. Rhodes, Governor of Ohio, cites as social 
ills (unachieved social values) unemployment, welfare, and 
lack of skills. He urges support for occupational educa- 
tion as a means to address these social ills (Rhodes 
1969, pp. 13, 16) . With a different emphasis from Prosser, 
though not inherently contradictory, he states that voca- 
tional education gives definite purpose and meaning to 
education by relating it to occupational goals. "It 
provides the technical knowledge and work skills necessary 
for employment, but it is more inclusive than training for 
job skills. It develops abilities, attitudes, work habits, 
and appreciations which contribute to a satisfying and 
productive life" (p. 44). 

"Vocational education contributes to the general 
education needs of youth, such as citizenship, respect for 
others , and acceptance of responsibi li ties; but , " says 
Rhodes, "it makes its unique contribution to the field 
of work" (p. 44) . 



Bell, in 1975 , in testimony bef':^re the Conoittee on 
Education and Labor, acknowledged the legitimacy of ab--.e3si 
vocational education, in pr> t, by the match between 
vocational education programs and employment opportunities. 
He maintained, however, that it woild be an error to judge 
■-he success of all vocational oducatiou by this criteria 
alone. "We feel" he stated, "that vocational education 
programs should be perceived as an integral part of the 
educational system of the country . . . and as such is 
responsible to assist in increasing and improving basic 
cognitive skills, heiqnteninq career awareness, improving 
the understanding of a variety of work environments and in 
many instances actually motivating students to remain in 
school at the secondary or pos "secondary level as well as 
providing specific occupational skills" (pp. 308-399). 
Ho added that the two criteria of measuring success in 
vocational educational procjrams by ratio of program 
completions to enrollments and the employment rate of 
graduates lose sight of these other aspects of vocational 
education. 

What are the commonalities and differences between 
these expectations of leaders in vocational education and 
rep"r«^senta tives of the policy and one consumer of voca- 
tic^/ial education namely the employer? One noted authority 
on business and management provides one source of in- 
formation. Drucker ( Management , 1974, p. 267) brings 
together social and individual values when he describes the 
prerequisites for responsible workers and parenthetically 
states that the fundamental reality for every worker is 
the eight hours or so he/she spends on the job through 
which the great majority has a ccess to acnievement, to 
rultillment and to .--ommunity. In order to achieve, the 
worker must be ablt to take responsibility for the job. 
To do this requires: (1) productive work; (2) feedback 
information; and (3) continuous learning (p. 267). Pro- 
ductive work is based on knowledge, analysis, and skill. 
Feedback information about the worker's performance as 
against standards is the major reinforcer and tool of the 
worker for measuring and directing herself/himself. 
F^elative to continuous learning, nrucker notes that 
continuous learning does not replace - tra ining . "I has 
different aims and satisfies different needs. Above all, 
;.t satisfies the need of the employee to contribute what 
he himself has learned to the improvement of his own 
performance, to the improvement of his fellow worker's 
f)er f ormance , and to a better, more effective, but also 
m^i*e rational way of working." Continuous learning 
comes to grips with the two basic problems of resistance 
of workers to innovation, and the danger that workers 



will become obsolete. It is as appropriate for clerical 
work as for manual work and knowledge work. The work 
group has to be seen and has to see itself as a learning 
group (p. 270) . 

A survey of employers in New Jersey was reported in 
197 6 ( Task Force on Competency In dicators and Standard s , 
Rutgers Research Team, October 1976) as a part of an 
effort to identify minimum basic skill requirements 
for employment for high school graduates . Small , medium, 
and large f i rms were surveyed to explore the area of employ- 
ment Practices to determine certain basic parameters. It 
was found that " reauirenents for employment were most 
typically job--related and are becoming more so as a result 
of recent court rulings. Very few aeneral academic 
requirements were found although some firms indicated they 
[^referred high school <:;raduates particularly for clerical 
positions. For some skilled occupations ^rade school 
training and /or apprenticeship training is required . High 
school graduation was of ten seen as a measure of d spend - 
ability and ' stick- to-it-i\"^ness ' rather than as a 
guarantee of basic skill r^-^tery" (p. 14). 

Cc". Tiunications and arithmetic were coiiunonly stated 
requirements '"or clerical positions and were considered 
important for a wide range of jobs as well. Firms using 
tests are more likely to be large and to have specified 
requirements for each job. Such tests are practical and 
job-related as opposed to general academic tests employed 
in the past since courts have ruled that tests must have 
demons tra'-od validity for the job. 

In sum, coiTunonali ties do exist among these producers 
and consumers of vocational education/ howeve; differences 
on some points of significance appear--specif- lly dif- 
ference in the narrowness versus the breadth of program 
objectives; in one sense narrow training versus education 
for occupational activity; the explicit attention to 
educating in processes that have more enduring value as 
we 11 as the immediately relevant job specific skills. 
Taking into account the customary gap between ideals and 
practice , the narrowly conceived outcomes i 1 lustra ted by 
the GAO study, the e: ,jhasis in the legislative mandates 
for preparation for employment, and the more frequently used 
outcome measure s of j ob olacement , length of time to gain 
employment/ and employer satisfaction, one can speculate 
that the narrow view is in all likelihood more pervasive, 
more prevalent, and more predominate in program implementa- 
tion as well as evaluation. 
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The narrow expect jLt lor.s rioem based most closely on the 
values of efficiency and rjracticali ty m the short term. 
Whereas the devt.» 1 oumer. of "occupational intelligence" 
in Prosser diid Allon's term attend to achieving values of 
efficiency and ;;ract ica i ity over the long term, individual 
freedom, intririSic valuinr; of individual personality, and 
facilitate achievement ".nd success through work. 



Parents and Students: C ur. suners Wi::h Val ; " s 

Rokeach (1973) defines values as the cognitive 
represeiitat ions and transf ormaticns of individual needs 
and societal and institutional demands. He thus clarifies 
and adds a dime ns ion to Ma slow *s hierarchy of needs as 
motivating forces as he : - oceeds to differentiate terminal 
and instrumental values. I'erminal values are end states, 
while instrurr.eiU al values vire modes of behavior some of 
wh ich are mora 1 va 1 ues , others of which are competence 
values. Value systems refer to the ranking of terminal 
values ale. a continuum of importance; instrumental values 
are viewed as being ranked also. Rokeach found that adult 
men and women ranked th^, following terminal values highest. 



Rank 



Men 
665 



Women 
744 



1 
2 
3 
4 
5 
6 



Wor id at Peace 
Fami ly Secur i ty 
Freedom 

Comfortabl ^■ Li fe 

Hap[jinoss 

Se 1 f-res: '-ct 



World at Peace 

Family Security 

Freedom 

Salvation 

Happiiiess 

Self-respect 



i i^.striimental valuers I'ar.kuci as the top six were: 



Rank 



Men 
66S 



Women 
744 



1 



Hones t 
Amb 1 t LOUS 
Res pons i bie 
Broadminded 



Honest 
Forgiving 
Respons ible 
Ambitious 
Broadminded 
Courag •: "^iis 



3 
4 

5 



Courarjeous 
Fc;r: ( L ^' i ng 



o 



As Williams noted, and Rokeach implemented, detailed 
descriptions of values must take into account various 
social categories such as income, education, race, and age. 
An analysis of values compared by income and then by 
education found that persc^ns of low income and limited 
education held r.uch the same values some of which differed 
from those with high income and higher levels of education. 
A world at peace, family security, and freedom ranked high 
rather consistently across all categories. A comfortable 
life, self-respect, and friendship tended to be ranked higher 
by those with lower incomes and less than hicih school 
education than were ranked by those with liicjhcr incomes 
and who were high school araduates and above. Instrumental 
values ranked high by those with less than high school 
education were: honest, clean, ambitious, forgiving, 
helpful , and courageous . Being responsible was ranked hicih 
by those hr=«^'ir. .■ completed some high school and above. 
Relative t..^ rac, when income and education were held 
constant, (-.he cne major difference was the higher ranking 
accorded equality, second for blacks and twelfth for 
whites (Rokeach, pp. 57-72). 

For these data to have any import for the concerns of 
this paper, it is necessary to make several assumptions. 
First we must assume that Rokeach ' s data have some limited 
generalizabi li ty . Second, we must assume that m.en and 
women look to occupations as one means for achieving these 
values, with the corollary that vocational education 
programs interpreted in terms of these values would be 
more highly valued as a means. 

If these assumptions are at least reasonably tenable, 
then the findings have some utility for conceptualizing 
outcome measures which relate to the efficacy of vocational 
education in achieving aoals derived from these values. 

In an effort to gain insight into the values of secondary 
age students, the above data on adult values have utility 
only if we can accept the premise that these adults as 
parents, will desire for their children some of these 
same values. If so, then interpretations of program 
characteristics , goals , and evidence of outcomes could be 
developed with close attention to the conditions and 
behaviors which are associated with attainment of these 
values. The recent emphasis on educating in life skills 
for productive participation in society, as a high school 
graduation requirement lends some support to such a premise. 
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As for secondary aqe youth, themselves, Rokeach provide 
some data, inconclusive and limited however, regarding the 
salience of particular values for different ages at dif- 
fereiit developmental staqes. Terminal values that 
aenerally are ranked in the top eiqht by eleven, thirteen, 
fifteen and seventeen year olds are: a world at peace; 
freedom, family security (lowest for seventeen year olds); 
friendship; equality; happiness; and a comfortable life. 
Relative to instrumental values, honest; loving; forgiving; 
rank high for eleven, thirteen, and fifteen year olds. 
• ScTie significant departures between eleven year olds, 
thirteen, fifteen and seventeen year olds occur with 
increased value placed by these older qroups on being 
ambitious, and responsible and then for rhe fifteen and 
seventeen year olds, beinq capable and independent. 

N^aslow's hierarchy of sets of needs may be useful 
in analyzing some of the stronger needs in relation to 
s\jccific aae aroups of students. 

These sets are, in order of lower and higher needs, 
[jhysioloqical; safety; belongingness and love; esteem; 
and self-actualization. Examined in relation to Rokeach's 
findings, we find some congruence between terminal and 
instrumental values and the first four sets of needs. 
W^ith the fifteen and seventeen year olds moving nearer to 
that age, where a cireater dearee of economic self-reliance 
is socially approved, terminal and instrumental values 
come into play that contribute to meeting one's own 
[jhy sioloqical needs and safety, as well as enhancing esteem 

/'My work on values emphasizes the complex interaction 
between value complexes as motivating forces in choosing 
amonq alternative courses of action, objects, or goals. 
Hence to treat terminal and instrumental values as singular 
even for purposes of analysis, introduces the hazard of 
dangerous oversimpli f ication . For in the real world of 
action, conflict between value sets can and does occur, 
and various combinations of choices can lead to a potential 
optimizing of satisfactions related to several values. 

As a useful illustration, let us consider the generally 
low status of vocational education, A status conceded 
and deplored in such statements , frequently quoted , that 
we must eliminate the belief, too frequently held, "that 
vocational education is for other people's children." 
A proposition which is accorded the status of almost, if 
not quite, beinq a sociological principle is that the 
status of a service croup, e.g., educators, social v/orkers, 
physicians, refiects to some degree the status of the 



group served. Hence, teachers prefer to teach children 
from affluent homes; physicians are ^lore numerous in 
affluent communities; and social wcv'^orc, serving the poor, 
and disenfranchised have relatively lr\7 status among 
professionals and salaries reflect 1^- s. Vocational educa- 
tion prepares individuals for occupations that tend not 
to be lauded in the highly visible world of the mass 
media. Indeed with the differentiation of work and its 
being closeted in plants, factories, and offices, general 
knowledge about these occupations, as well as some higher 
status ones, is relatively sparse except for those areas 
in which friends and relatives are employed. The fact 
that secondary youth have been well socialized to this status 
dimension was demonstrated in studies of adolescent choices 
for work. Several researchers concluded that based on the 
distribution of workers in the work force, the education 
and ability levels required , the choices of large numbers 
of adolescents had to be labeled, in the aggregate, as 
unrealistic since professional work was cited by large 
numbers. For the field, at large, the relative status may, 
in part, contribute to the lower level of support from 
federal sources when compared with support for higher 
education relative to numbers served, although it is 
vocational preparation for what are generally higner status 
occupations . 

If we review the major value orientations delineated by 
Williams, such as achievement-success; activity-work, 
material comfort, and freedom through personal autonomy, 
I believe that the degree of attainment of these values in 
generally accepted social terms are optimized in some of 
the higher prestige occupations. Perhaps even more 
important is that most people implicitly accept the 
premise that the best optimizing occurs though involvement 
in higher status occupations. To the extent that these 
are valid observations, it follows that in choosing voca- 
tional educatici as the means for attaining some of these 
social values manifest as personal needs and values, many 
individuals have moved from the most preferred choices to 
the less preferred choices. An important qualifier to 
all of this, however, is the proposition that membership 
in socioeconomic groups tends to place a ceiling upon 
aspirations, and has been viewed as a social phenomenon 
that helps to reduce extreme feelings of frustration and 
deprivation. Thus, parents and family friends continue 
to be role models and prime influences in choice of and 
employment in occupations. Even given this qualification, 
there is evidence that a number of parents derive less 
pride from their children being enrolled in vocational 
education than in their continuing in a general or college 



preparatory curriculum with littlo ur no prospect of 
further education in the offing. Given these conditions, 
there would appear to be much merit in interpreting the 
value and outcomes of vocational education, as means tc 
these valued ends, quite explicitly for parents and 
students. There is, I suggest, too little mention of the 
likelihood that vocational education for some is preparation 
for work which v/ill be transitional and in turn a means to 
other statuses. Such an a;jproach may engender sc ,;ie risks 
given the GAO and supf^ort m some quarters for the position 
that vocational education can 'be justified almost ex- 
clusively by job placemen t--and that in occupations for 
which trained, or in a related occuDati("n. 



Ou tcome Measure s in Ev a lua t ing Vocational Education 

Evaluation and ovai uat i vo research need to be dif- 
ferentiated for fruitful consideration within the param- 
eters of this paper. Suchman (1967) proposes the "dis- 
tinction between evaluation as the general process of 
judging the wor thwhi leness of some activity regardless of 
the method employed and evaluative research as the specific 
use of the scientific method for the purpose of making an 
evaluation" (p. 31). He continues that the range of 
variation can be indicated by "defining evaluation as the 
determination (whether based on opinions, records, subjective, 
or objective data) of the results (whether desirable or 
undesirable; transier.t or permanent; immediate or delayed) 
attained by some activity . . . Designed to accomplish 
some valued goal or objective (whether ultimate, inter- 
mediate or immediate, effort or performance, long or short 
range)" (p. 32). Suchman points out that the evaluation 
process moves from, value formation, to goal setting, to 
goal measuring, ident i f y i.n<f goal activity, to putting goal 
activity into operation, to assessing the effect of this 
goal operation, and back to value formation. 

Specifically, what values are addressed in the outcome 
measures recommended for use in program evaluations and 
system, evaluation, derived by aggregating outcomes from 
program evaluations? The ::;ource for these recommended 
outcome measures is "A System foi' Statewide Evaluation of 
Vocational Education," The Center for Vocational and Technical 
Education. In broad categories, these outcome measures 
include: (1) descriptions of characteristics of individuals 
served by the program and, in aggregate, the system; 
(2) the successful completion or early departure from the 
system; (3) accept ance--e:<clus ion rates; (4) employment 
and earnings history and current status; (5) aspirations 
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for further education; and (6) satisfaction with program. 
Among the major value orientations evident in the choice of 
these measures are: attainment of work versus unemployments- 
earnings are one indication of level of material comfort, 
achievement and success; satisfaction levels and interest 
in further education acknowledge the incrinsic worth and 
a degree of personal freedom through autonomy of choice 
ard further development; admission, exclusion, character- 
istics of persons served, length of involvement, and conditions 
of departure address the value of equality of opportunity 
through evidences of accessibility. Attention to the means 
value of efficiency and practicality underly the data 
collected regarding program length, use of advisory 
councils for evaluation, current manpower data for program 
planning, facilities, and inclusion of cooperative work 
experience or simulated work experience . 

Statewide system evaluations are supplemented by program 
and local district evaluations. Some such efforts have 
included measures of attitudes towards work, work-related 
behaviors, self-concept, source of interest in work, 
changes in self-evaluation in relation to work (Nelson and 
Jacoby 1967; Kievit 1973). These outcome measures address 
more specifically the major value orientation of freedom 
through personal independence, self-confidence, and autonon.y, 
and the intrinsic worth of individuals as warranting develop- 
ment in the broader aspect of Prosser's "occupational 
intelligence." These are at the most elementary level, 
however, and limited rather than comprehensive. Although the 
number of sources of outcome measures used has been 
limited, this primarily grows out of my immersion in 
evaluation in vocational education which leads me to believe 
that these outcome measures accurately illustrate the 
range of measures most frequently used. 

Outcome measures which address individual values 
related to freedom, achievement and success defined in 
individual terns relative to aspiration levels, and 
individual personality such as of intrinsic worth are 
employed in systematic evaluative research less frequently 
and in more limited evaluative studies. These are more 
likely to be addressed in evaluation which includes heavy 
reliance upon informal observation, subjective assessments 
of spontaneous expressions of students, parents, and 
employers. Evaluation at this level is more likely to 
occur without explicit awareness of the full range of 
values which could potentially be furthered in attainment/ 
and with little attention to the nuances and shadings in 
communication with members of various constituencies which 
point to the values most salient to that person at that 
time. 
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The value of the initial distinction between evaluation 
and evaluative research resides in facilitating greater 
clarity regarding the purpose of evaluation in a specific 
situation, and the intended consumer of the report as this 
has implications for choice of outcome measures and data 
to be collected. Administrators, program planners and 
C'valuators, and teachers should be a team of producers and 
consumers of evaluative research. This team should be used 
as a basis for fine tuning the organization of educational 
experiences to provide the optimal outcomes in terms of 
manual skill, technical knowledge, and occupational 
intelligence. In addition, administrators, counselors, 
and teachers are strategically placed for explicitly 
linking program objectives and outcomes with individual 
values of studen ts and their parents . Administrators at 
the local level and particularly those at the state level 
need to be attentive to interpreting the outcomes of 
system evaluative research in terms of valued means which 
characterize vocational education, and the valued ends to 
which vocational education contributes. 

The choice of outcome measures should be reexamined and 
expanded to more specifically seek information regarding 
efficiency and practicality not only in the short term but 
also over the long term. The contribution which vocational 
education has made to the individual's sense of attaining, 
to some measure, the values of family security, a comfortable 
life, a sense of expanded freedom through expanded options 
among which to choose, with a sense of confidence, self- 
respect, equality, and social recognition should be docu- 
mented. The issue is not whether these values might not 
have been attained by other means--they probably would have 
to some degree but rather to demonstrate that vocational 
education is one means among others, equally effective in 
most cases and more effective than other means in some 
cases. This type of docuraentat ion would be particul'irly 
rich as a basis for information to disseminate to parents 
and potential students, as well as legislators concerned 
about the opinions and views of their constituencies. 
Interpretation of outcomes to this last group might be more 
effective if couched in terms of equality of opportunity 
through differentiated and expanded options for many: 
the sense of success by virtue of having the capacity to 
purchase more of the symbols of success, through higher 
earnings, even if the work is mid-range to low in prestige. 

Documentation of this nature would seem to call for 
tracking of vocational education students through time, at 
various regular intervals. The case study method with 
interviews would afford the greater likelihood of gaining 
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insight in the contribution of vocational education to 
value attainment. The diversity of the life styles surfacing, 
the interplay between work and education, job changes with 
impact on value attainment, would be informative and add 
much to our understanding of the impact of individual 
programs and state systems. 



Legislators and Board Members; 

Audience for Evaluative Research and Evaluation 

Legislators and board members are concerned about 
accountability to their various constituencies. One criterion 
for accountability is her/his effectiveness in promoting 
arid sustaining federal, state, or local policies and programs 
which are perceived to be beneficial. Benefits are related 
to the achievement of goals linked to values. In view of 
the preceeding examination of values, it seems reasonably 
evident that values most salient for these groups include 
an employable rather than unemployable constituency, 
constituency material comfort vs. material impoverishment, 
independence of individuals vs. dependency, and equality of 
opportunity, and equal access to programs. Credible 
documentation that vocational education is contributing, 
efficiently and practically, to attaining these ends for 
an important segment of a legislator's constituency will 
provide a more persuasive rationale for support. Evaluative 
research and evaluation should be the basis for providing 
this documentation . 

E\*aluative research should include the placement, job 
satisfaction, admission and completion data elements as 
well as the others cited earlier. It should be supplemented, 
however by some systematically obtained case materials which 
provide "close-ups" of what the data mean in individual and 
human terms. Data obtained from case studies should be 
targeted specifically on the values of individuals enhanced 
by vocational education. The results should be couched 
in "value" terms when reported. 

Report format might be modeled (with some adaptation) 
after the annual financial reports of a large corporation. 
The better ones of these show the statistical data succinctly, 
and elaborate on the meaning of these statistics for the 
central concerns of the firm. The narrative could include 
the "close-ups" of individuals and groups served with some 
general statements indicating the typicality of the 
"close-up" for the total population served . Goals only 
partially achieved, but the subject of imaginative planning 



and continued pursuit, should be included and clearly 
presented as evidence of shared concern for efficiency 
and practicality as a means to the end values. 



Employers: Audience for Evaluations 

Employers value productive workers as essential to 
increased efficiency. Hence, evaluations should attend to 
those program outcomes which are linked to student behaviors 
essential for rapid integration into the job setting with 
full productivity within the shortest time period feasible: 
habits of work, such as knowledge of safety measures, 
dependability that reduces loss of time , a f airlv precise 
indication of level of skill performance to be expected 
from specific vocational programs, and the adaptability 
and capacity to learn is essential for responsible workers. 
The values of employers which vocational education contribute 
to are comparatively narrow. 

Modes of reporting might well be in the form of an 
inventory of these behavioral outcomes from vocational 
education programs. The proportions of graduates who 
achieved minimum levels, clearly defined, in the various 
areas could be reported in such a format. 



Administrators and Te achers : 
Audiences for Evaluative Research 

Evaluative research should be designed to link assess- 
ments of outcomes to the attainment of individual values of 
students, both present and emerging. Close attention needs 
to be given to the controllable variables which may be 
significant points of intervention for teachers in struc- 
turing curriculum, classroom management, and instructional 
strategies to increase the efficiency of learning for many 
students , increasing the success rate for those admitted , 
and enhancing attainment of the personal values of students 
for (a) feeling capable, (b) achieving a sense of esteem 
from self and others, (c) sharina friendship, and (d) 
increasing individual freedom through decreaseu dependency 
and increased capability for independence in a n^jmber of 
spheres through the attainment of skill, problem solving 
abilities, and earning potential. Setting goals, and 
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monitoring progress or lack of progress toward these goals 
are essential to evaluation of ongoing program implementation 
designed from evaluative research. Routinized feedback 
sessions between supervisors and teachers on cxjraulative 
program outcomes for students should be an important part 
of the process. The feedback should include summarizing 
quarterly reports and an annual report. Some quantifiable 
data of use would include attendance rates, dropouts, 
measures of skill performance in vocational area, anecdotal 
observations regarding problem-solving skills and those 
behaviors indicative of goal attainment relative to the 
va lue s ci ted above . 



Students and Parents 

The values of individuals to which vocational education 
is one means include family security, freedom through 
economic independency, self-respect, and material comfort. 
Work and preparation for work are processes through which 
individuals develop and exercise friendship, responsibility, 
capacity and ambition. Evaluation which is couched in 
"close-ups, " vignettes , and anecdotal types of communication 
can effectively link outcomes of vocational education to 
individual values. A series of well-designed, colorful 
posters, each focusing on a single value and multiple program 
processes and outcomes which are related is one reporting 
format. Spot public service announcements on raiio and 
brief film clips on television designed on the same rationale 
could be more widely disseminated. News releases in local 
daily and weekly newspapers leading off with a "close-up" 
for human interest on one or two values and concluding with 
statistical data regarding the typicality of the achievement 
could also be effective. The focus and reporting made 
could be similar for parents and students with slightly 
different points of emphasis. Young students in secondary 
programs are less concerned with f ami ly security but are 
concerned with acquiring freedom through independence, 
being capable and developing friendships. Data obtained 
through case study and interview are best suited for 
dissemination to these groups . 

In conclusion, as vocational educators concerned with 
linking programs and program outcomes more closely to 
related values of our various audiences, we should direct 
attention to ways and means of ascertaining those values 
of persons and groups of immediate concern . This paper 
provides a general framework, a spring board, so to speak, 
from which to be better attuned to appropriate starting 
points in seeking information on values. It is the broad- 
brush stroke and not the finely detailed picture needed 



for most effectively linkinq :.^-^.I-t j.cular programs in 
vocational education ro \'3luGS--social or individual. 
To seek such Imkaaes is a worthwhile direction. For 
in the process, individual rurposes and legislative 
intent may be more clearly articulated, more subject to 
reexamination and prosit ivo nodi fi cat ion. Vocational 
education nay become noro 'jffoctive as a means to attaining 
social and personal \'alues. 
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OUTCCffi Em:S ffiLATED TO INDIVIDUAL AND SOCIAL VALUES 





Individual 




Major Americin 


Outcome Measures 


Termnal 


Process 


Individual Needs 


Value Orientations 




CooifQi;t.il:'le life 


/imbitious 


Physiological 


Achievement-success 




Sense or ac- 


Capable 


Safety 


Activity-work 


Work - Earnings 


compliiiraGnt ' 


Honest 


Esteem 


Material comfort 


Freedori 


Independent 


Self-actualization 


Freedom through in- 




Self-respect 


Logical 




dependence, self- 




Social 


Responsible 




confidence, per- 




recognition 


Self-controlled 




sonal autonomy 
Efficiency and 
practicality 


1 Access, enroll- 


Fqeality 




Belonging 


Equality of opportunity 
Humanitarian mores 


^ raent and 
1 completion 


Froedo?, cf 
choice 





Esteem 


Efficiency-practicality 
Democracy 

Individual personality 


Aspirations 
for Further 


"aterial comfort 






Achievement-success 


Sense of 
accomplishment 


:\mbitious 
Capable 


Self- 
actualization 


Efficiency-practicality 
Material comfort 


Edxation 


Freedom - Clioice 

Independence 
Self-respect 






Equalitv of opportunity 
Freedom 


Satisfaction 


Pleasure 






Humanitarian mores 


with 


Happiness 






Efficiency and 



program 



Program 
Cnaracteristics 

- Work experience 

- Manpower data 

- Advisory councils 
o - Length 



practicality 
Individual personality 



Efficiency and 
practicality 
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REACTION TO PERSPECTIVI SM IN CHOOSING 
AND INTERPRETING OUTCOME MEASURES 
IN VOCATIONAL EDUCATION* 



Some of you know that I live in the Washington, 
D. C. area; well, I must say it's good to be away from 
Disne\ ^1-Three. Maybe in the calmer setting of Louisville 
we r-' p away at some of the thorny issues which have 
pl -ocational education since 1968. To me, 1968 

was L iiv. Lime when some observers began to take the voca- 
tional education movement seriously . Traditionally , 
vocational educators were more concerned with day- to- 
day activities and left others to agonize over items of 
theory, relevance, and outcome measures, for example. 
Since 1968, the critics have advanced in numbers and 
become more strident in their comments. In addition, 
other events occurred which have given comfort to those 
who would do away with or diminish vocational education. 
To illustrate, the apparent anti-tax movement in California 
caused the dissolution of RAVACs (Regional Adult-Vocational 
Advisory Councils) after a two-year development period. 
Another situation is the back-to-basics concept. Generally, 
vocational education is not considered a basic by the 
proponents of this idea. 

These situations tended to make some vocational educa- 
tors defensive, others angry, and still others frustrated. 
However, a variety of activities which are designed, in 
part, to resp ^^.d to the issues are occurring. In a sense, 
this conference was brought about by the need for constant 
thesis, antithesis dialogue. 

Mary's paper makes a significant contribution to the 
dialogue. It splendidly renews the memory of our heritage. 
It is a refreshing and stimulating reminder that the 
largest part of what we are all about is people and their 
needs ; not 0MB ' s economizing . 

More specifically, I would like to draw your attention 
to four possible applications of some of the thoughts 
that Mary provided for our consideration. The four areas 
where consideration of values can have a major effect are: 
(1) Philosophy of Vocational Education, (2) Curriculum 
Development, (3) Research, and (4) Management. 



Reaction by Ralph Bregman to Mary Kievit's presentation. 
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Philosophy 



In 1977, Floyd McKinney, Katy Greenwood and others 
conducted an excellent workshop on the philosophy of 
vocational education. The workshop itself and the number 
in attendance were tangible evidence of the interest in 
philosophy — the theory as well as the practicality. 

Those who are continuing this vital effort — the 
development of a philosophy — would do well to think 
about the values of vocational education as part of any 
statement or pronouncement. 



Curriculum Development 

Ifr in fact, vocational education can satisfy 
reinforce certain behaviors which are cognitive/aft 
domain-related and which , in turn , adhere to selected 
values r then the content designed to develop behaviors and 
cultivate values should be specified in the curriculum. 
Too many of the alleged benefits of vocational education 
cannot be identified in the curriculum. 



For example r at a recent meeting, a review of the 
literature on vocational education was presented. The 
speaker noted that vocational education has multiple goals 
Among its many claims is that it provides personal satis- 
faction to the individual; it also promotes social and 
economic mobility. The curricula that I ' ve seen have no 
content referent to these goals. The goals and the 
individual's achievement of same seem to be assumed as a 
result of participation, per se, in a vocational program. 



Research 

Mary has provided us with a taxonomic paradigm for 
linking, as she says, particular programs in vocational 
education to values — social or individual. 

I would like to suggest a more ambitious use of these 
materials. The vocational community has an opportunity 
to put forward and restate in specific terms the social 
and economic utility of vocational education to the 
individual and society. Put another way, the values that 
relate to vocational education could be restated as 
intended outcomes. A matrix could be prepared. On one 
dimension , social and economic goals that vocational 
education purports to devel^ p could be noted. Individual 



and societal preferences related to vocational education 
would be on the other dimension. (The cells on the matrix 
for any particular qoal or preference need not be mutually 
exclusive.) For example, a social goal espoused by some 
vocational educators is the individual's attainment of a 
firmer, more positive se 1 f -concept . Self-concept is 
congruent with "individual personality," a value orientation 
identified by Williams. Where an individual's preference 
is congruent with this qoal, then the results of curriculum 
intervention can be reported in more precise terms . 
Currently , results are presented in a unidimensional form — 
that is, how many students were enrolled or how many were 
placed . However, if values are added to the research, 
then the outcome modality, in the form of a multidimensional 
display, can be achieved. 

Additionally, by dc; , , more specif. AV^.i 
vocational education is wi . . ng and preparea uo be account- 
able for, those enga(jed in research will not be able to 
operate as loosely as heretofore. One illustration should 
suffice: In a recent study (April 1978) reported in the 
Phi Delta Kappan , one statement made by the author was 
that vocational education programs, as currently designed, 
appear to alienate students from themselves and society while 
failing to engender any kind of appreciation for learning. 
In reviewing the original manuscript in order to prepare a 
response, I could not find the criteria that the author 
used to classify students as vocational. Further reading 
disclosed the author's [)Osition, which was that "the nature 
of vocationally oriented curricula is so diverse as to 
render any clear distinction between academic and vocational 
education nearly impossible." Unless the vocational 
community stands forth and more clearly identifies itself, 
it v/ill continue to be ^^laqued by irresponsible interpreta- 
tion of vocational proqramming . 



M anagement 

Management o f the vocational enterprise can be 
influenced by the vocat i ^.)nc^l values held by administrators. 
An educational managtjmen t system generally includes five 
components: information, planning, operations, evaluation, 
and reporting. Each of these components has a vital role 
to play in perfecting the vocational delivery system. 
Personnel managing these components who have not internalized 
vocational values would be only superficially concerned with 
th • value structure imparted to students. These same in- 
dividuals have avoided respons ibi lity for deve loping a 
universally accepted definition of a vocational student. 
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Are we really content in asserting that a high school 
student who took only one vocational course was a bona 
fide vocational student? That is, that she/he ^acquired the 
requisite social and economic skills and values. If we 
are not content, why do we report them as vocational 
students rather than prevocational or by some other more 
descriptive term? 

In conclusion, I hope that Mary's paper acts as a 
catalyst and causes us to consider the broader applica- 
tion of values — to put to use the value structure imbedded 
in the vocational system and of those it serves and report 
the outcomes of some activities in terms of societal/ 
individual values. Further, in order to obtain data of 
any reasonable magnitude of the impact on the value 
structure, the societal values addressed by the different 
subsystems (e.g., curriculum development and research) 
must be explicated. Finally, if we consider carefu"' ' ' 
tht '.J ■ ^ implications of her message, we might fin. a; 
vocational education will be better able to take a n^uc 
proactive and aggressive posture. 
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REACTION TO PERSPECTIVISM IN CHOOSING 
AND INTERPRETING OUTCOME MEASURES 
IN VOCATIONAL EDUCATION* 

Let me first express some hesitation in proceeding 
with my prepared remarks on Dr. Kievit's paper. In my 
first and second readings of the advance copy of the paper, 
I came away with differing impressions. In hearing her 
presentation today, 1 discerned other new ideas in the 
paper. Probably, when rereading it again later, I will 
discover yet additional new concepts in what Dr. Kievit 
has produced. So it is only with trepidation that I 
proceed to offer my : ropared remarks. 

Primarily, I looked for certain things in the paper 
that, unfortunately, I was not able to find. My expecta- 
tions for the paper had been based upon discussions and 
concerns expressed during the meetings of the Advisory 
Panel for this project. I am sorry that the paper does not 
address these concerns. 

The three-fold purpose of the paper is to describe the 
values held by various groups; to discuss the roles played 
by values in the groups* views about vocational education; 
and, finally, to consider the implications of the foregoing 
for the choice and interpretation of outcome measures 
used in evaluations of vocational education. 

One group addressed in the paper is comprised of voca- 
tional education leaders, and Dr. Kievit cites Charles 
Prosser, James Rhodes, Tcrrel Bell and others. She points 
out that the leaders suggest that vocational education 
contributes to the attainment of a wide variety of valued 
social and individual goals , including : 

• efficiency in the provision of occupational 
training 

• securing stability and progress in a democracy 
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promoting morale and intelligence among workers 



• combatting unemployment and reducing the need 
for public welfare by providing skills 

o developing attitudes, abilities, work habits and 
appreciations which contribute to a satisfying 
and productive life 

• contributing to citizenship, respect for others, 
and the willingness to accept responsibilities 

• assisting in increasing and improving basic 
skills, heightening career awareness, and motivating 
students to remain in school 

Unfortunately, her paper does not addros* the in>pli» i 
tions of these ^' ou^; Valued goals for t^. ;j.ce and 

interpretation ot uutcume measures. However, it should 
seem that there would be important implications. 

It would seem to me that research undertaken or 
supported by vocational education leaders in the states 
cind LEAS should reflect the values and goals listed by 
Dr. Kievit, and that findings should serve to document the 
accomplishments of vocational education. But the paper 
does not address these points. 

Moreover, as I considered the body of existing research 
and evaluation studies, I see possible reasons why the paper 
did not consider the implications. First, the valued 
goals may be so fuzzy, global, and difficult to measure 
that it is impossible to identify credible evidence on 
the relevant accomplishments of vocational education, 
because no such evidence has been produced. An alternative 
possible reason is that there is credible research and 
evaluative work that relates to the values and goals in 
question, but that Dr. Kievit chose not to review and 
discuss it in the paper presented today. Let me insert 
a sympathetic note: A good deal of work done by^ and for 
State Departments of Vocational Education and by and for 
State Advisory Councils for Vocational Education is very 
difficult to learn about and to obtain. Frequently such 
work is not regularly submitted to the ERIC system for 
dissemination , nor to NTIS , nor even to the National 
Advisory Council on Vocational Education for use in 
connection with their advocacy role . Even though this 
body of work may find use within a state or local area, 
it should be clear that whenever Congress eind other relevant 
audiences do not know of its existence, then for other 
purposes it might as well have never been produced. 
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Legislators comprise another group addressed in the 
paper, but I believe that the discussion might have been 
improved. For instance, one section of the paper refers 
to "narrow" (versus "broad") perceptions of the role of 
vocational education. This portion of the paper refers 
specifically to the emphasis in federal legislation to 
matters of: preparation for employment, job placement, and 
employer satisfaction with graduates of vocational educa- 
tion programs. In doing so the paper conceives of the 
legislative mandate for evaluation as if it were unduly 
narrow. It seems to me that such an approach fails to 
appreciate the perspective that such legislative provisions 
are completely consistent with at least one of the legitimate 
sets of values based on which ^' --n' Jona1 oHucai-i- • 

Another group considered by the paper consists of the 
employers of alumni of vocational education programs, and 
the paper refers to values with respect to entry-level 
performance, as well as progression over the long run. 
This portion of the paper appears to reflect a dilemma. 
On one hand, employers are said to be sensitive to the 
long run. On the other, employers are increasingly forced 
to adopt "narrow" hiring criteria, owing to recent court 
rulings concerning the use of criteria not clearly related 
to immediate job performance. Employers may use "broader" 
criteria only after demonstrating their relevance, but the 
burden of proof rests with them. This dilemma may have 
implications in terms of employers' values for the planning 
and evaluation of vocational education programs. 

The paper also discusses values of adults, and of 
students as consumers of vocational education. However, 
the paper refers primarily to secondary-school-age students 
and does not explore the population of pos tsecondary and 
adult students f and how their values may differ from 
those of the teenage group. 

In any event, in summary I believe that this paper 
constitutes a valuable beginning. it is a credible 
scholarly treatment of "perspectivism" in the general area 
of vocational education. Specifically, I appreciated the 
paper's emphasis on the need for credible documentation 
of the outcomes of vocational education, and on the need 
to undertake evaluative research. 

It does not, however, seem well suited in my opinion to 
immediate use by states and local agencies in their 2.equired 
evaluation activities, for there are many questions that 
remain unaddressed. 
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The paper contains very little by way of review of 
existing impact research and outcome measures, and also 
of models and outcome measures related to various values 
that the author could recommend for future evaluative 
research. It does not indicate those types of outcome 
measures that relate to each of the important audiences, 
and does not address either the comprehensiveness or the 
adequacy of the measures now in use. 

It does not adequately address the valiies that jj' 
to underlie the I91f> fedor>i leai M^ ^: i t^^js 

embo^"'' -1 in - ■ . rlc r ...x xy-sponsored uvalua- 

ri^Miti-y uaUurway. Although the paper 
constitutes an important beginning, there is still a good 
deal more to be done. 
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EVALU . VO CATIONAL LbULAi lOl^ , 
A Ct-., ,HESS"I0NAL PERSPECTIVE* 



Tonight, I would like to talk about two things: 
(1) the context in which the new evaluation requirements 
for vocational education came about in the Vocational 
Education Amendments of 1976, and (2) the nature of the 
requirements themselves , 

I know you all are familiar with the history of voca- 
tional education so I wi 11 not review its deve lopment all 
the way back to the congressional enactment of the Smith- 
Hughes Act which initiated vocational education as we 
know it today. I will skip the first years and start with 
1963 when a landmark vocational act was passed. The 1963 
Act had the effect of broadening federal support for 
vocational education, and focusing more attention on it. 
The 1968 Amendments further broadened that Act and brought 
about some updating in programs through authorizing research 
funds and exemplary programs. These amendments also focused 
attention on special populations such as the disadvantaged, 
the handicapped, and on pos tsecondary students, whose needs 
Congress felt at that point in time were not being adequately 
addressed. Then, more recently, the 1976 vocational educa- 
tion legislation was passed which is why you are here 
tonight. 



THE EDUCATION AMENDMENTS OF 1976 

The 1976 Amendments build upon what happened in the 
past but they must also be viewed within their own context. 
Dur ing 197 5 and 1976 , Congress he Id very extens ive hearings 
and found that the data showed vocational education to be 
very successful, at least, in quantitative terms. There 
seemed to be many more dollars being spent at the state and 
local levels for vocational education than ever before. 
There seemed to be many more facilities, more students 
enrolled, and some empirical evidence that there was 
success with those students in the program. However, the 
legislators in reviewing the progress of the Act were 
confronted with several problems. 
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As I describe these problems, I -ruld like to emphasize 
that I am in effect summarizing the ju: ^ and Senate 
committee reports on this legislati > je committee 
reports are frequently the best, anc ometimes the only, 
documents which give a thorough explanation of the congres- 
sional intent surrounding legislation. 



PROBLEMS ADDRESSED BY THE 1976 LEGISLATION 

Isolation 

In 1976, these committee reports stated a general 
conclusion that vocational education programs as administered 
at the state level in many states seemed to have too much of 
a tendency to be isolated unto themselves. In state 
departments of education , vocational educators sometimes 
were isolated from other parts of the state department. 
Within the states themselves vocational educators were 
too isolated from postsecondary educators; and, also, 
vocational educators were too isolated from the people 
operating Comprehensive Employment Training Act programs 
and other types of training programs. And, of course, the 
same was true in reverse: these other people who ad- 
ministered programs very similar to vocational education 
frequently isolated themselves from vocational educators 
and others administering similar training programs. 



Use of Federal Funds 

An additional problem seemed to be that there was 
difficulty in finding how the funds were being used, 
especially the federal money. This was true for a variety 
of reasons . In many states , federal dollars were mingled 
with state and local dollars so that you really couldn't 
tell what the federal dollars were bringing about or even 
paying for. Although the state plans were supposed to 
show the expenditure of federal money, they really didn't 
show much because they were merely compliance documents 
completed to fulfill the exact letter of the law. There 
also seemed to have been a problem with the attitude of 
the federal administrators in that they operated the program 
as a revenue sharing program without going out to the states 
to give them assistance or trying to correct misuses of 
federal money that occurred. It appeared that generally 
federal money was being used properly, but possibly it 
wasn't being ur.ed properly in some isolated instances; and 
no one knew for sure how many of those instances there were. 



-106- 



Lack of Good Data 



Another problem seemed to be a lack of good data to 
show exactly what was happening, not only with federal 
iTiCney, but with state and local money as well. In one 
state, they counted anybody who was in career education 
as a vocational student, in other states they did not. 
In some states they counted students as vocational students 
if they were in one course for one hour a week, and in 
other states they did not. In some states you had to be 
a full-time student in a course for many hours, and in other 
states you could be a part-time student. Differences also 
existed across states as to the types of credent ialing 
used. There seemed to be no uniformity in the data, even 
sometimes within a given state. So, if you were to take 
the testimony of people who said that things were going 
well, you couldn't really find out for sure because there 
was not the sol id data base to back up those statements . 



Lack of Follow-up 

The last of what these committee reports called problems 
had to do with the inability to look at the quality of the 
programs. There were some witnesses who were able to show 
through follow-up studies that their graduates had done 
well. But this type of follow-up seemed to be very sporadic. 
You had somebody from a city saying: "We have this record 
that kids did well." But you really did not have any 
uniformity in the following up of those individuals over 
several years or data from other school districts in terms 
of whether that was true for different types of programs 
or even the same type of program over time. 



THE CONGRESSIONAL CONTEXT 

What Congress tried to do was to concentrate on these 
problems and bring about some solutions. At this point, 
I would like to repeat what the reports stated in describing 
these problems, namely that this description of problems 
was not meant to leave a negative impression of vocational 
education; rather, the Congress was trying to discover what 
the impediments were to a better vocational education system 
and was trying to help remove those impediments . 

It was presumptuous in a way for Congress to try to 
address these things because the federal commitment to 
vocational education, even though it has grown in dollar 
terms, has in no way been able to keep up with the e:.- 
pansion of state and local funding. Though state and 
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local money arew by 100 percent between 1971 and 1976, 
the federal appropriations increased by only 37 percent. 
Obviously, the federal government wad by far a junior partner. 

Nonetheless, Congress felt for several reasons that 
it should be bold in trying to say that certain things 
should be put into process. Congress was the first in 
trying to encourage vocational education in a national 
sense through enactment of the Smith-Hughes Act in 1917. 
And since then. Congress in this field peculiarly has been 
the body which has given some direction to vocational 
education, frequently out of proportion to the federal 
money which was being made available. Sometimes vocational 
educators look to Congress and are willing to accept some 
solutions even though Congress isn ' t providing the bulk 
of the money. 

Further, the friends of vocational education in Congress 
fe It that the ball game was being lost because within 
that same bracket of time when Congress was minimally 
increasing vocational education appropriations , bare ly 
keeping up with inflation, the appropriations for the 
Comprehensive Employment and Training Act (CETA) were 
growing by leaps and bounds. That program has gradually 
changed from a job retraining program as it was first 
constituted in 1962 to a public service employment program 
today with what seems to be a very low priority on job 
training. Yet, billions of dollars were going to the CETA 
program; and some Cong re ssper sons felt that vocational 
education had to show a better record in order to be able 
to receive some of that funding. The congressional friends 
of vocational education felt that a better case could be 
made for the training provided by vocational education 
because that training produced a long-term productive 
solution instead of a short-term reactive solution. 

In the Amendments of 1976, Congress tried to help 
vocational education make this case. First, more ad- 
ministrative flexibility was given to the states through 
"block-granting" most of the funds. But, then, second, the 
legislation concentrated on making changes in the way 
decisions were made by the states . In other words , the 
legislation concentrated on the decj sion-making processes 
at the state level . It seems that in vocational education , 
and, again, rather peculiarly, the state departments of 
education have quite a say in areas in which they don ' t 
always have a say in other types of programs. Consequently, 
Congress tried to impact on the decision -making at that 
level in four areas. 
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NEW AMENDMENTS 



P la nn iiuj Rcqui rements 

One area of clianqe involved the planning requirements. 
They were to be very specific. 7n this Aci , Congress 
was extraordinary in writinc/ out exactly how the planning 
process had to occur at the state level. The law specifies 
the exact agencies within each state which must participate 
in the Planning process. The law also says that there has 
to be so many meetings a year and describes what has to 
happen in each one of those meetings. That is extra- 
ordinary for a federal law. I think Congress felt a need 
to open up the decision-making process so that all elements 
of job training could get involved in the way in which 
federal funds were being spent. 



Ac cj^ untability 

Congress also required that the state plan, which was 
the document to be used for this decision making, had to be 
very precise and had to say exactly where the money was 
going, and most importantly why the money was going for 
certain thinns. Now I will dare you to go back and look 
at state plans which were compiled in vocational education 
for many years to find these things, and you will not find 
many plans setting out the exact reasons why certain things 
were done. It seems a lot of things that were in the law 
were simply regurgitated in the state plans; and that 
document, in fact, was not a planning document that presented 
reasons why things w^re dono or not done. 



Improvement of Data 

Another thing Congress Liiod to do was to include 
provisions in the legislation to enhance the chances for 
better data. In order to achieve better data, federal 
data gathering was transferred from the U. S. Office of 
Education to the TTational Center for Educational Statistics 
(NCES) . Ar.d, the legislation also laid out the specific 
elements of data which had to be collected by the states 
and from the states. Congress also mandated the creation 
of the National Occupational Information Coordinating 
Committee and companion state committees to help the states 
and NCES with this task. These committees were also to work 
on achieving more uniformity in data gathering among all 
training programs and to work on achieving the use of 
better occupational demand data in those programs. 
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Evaluation Requirements 



You are aware, of course, of the state evaluation 
requirements which are the last area with which the new 
legislation deals. Congress is saying that within the five 
years of its state plan each state has to evaluate all 
its programs. Evaluations of specific elements of programs 
offering entry-level training are required, i.e., as to what 
happens to the completers of those programs in terms of 
whether they achieve employment in job related fields and 
what their employers' opinions are of their abilities. 

I want to discuss why Congress chose those criteria 
and why it did not choose other criteria. I think there 
was a feeling that a program which claimed that it was 
training people to send them out into the job market should 
minimally show whether the people who were trained in that 
particular skill did or did not obtain employment in that 
field , and secondly , what the employer thought of their 
capabilities . 

There are many other things which could be taken into 
consideration because you cannot take human life and reduce 
everything to those two simple facts in order to judge 
"success" or "failure." There are always SvO many things in 
life, so many variables, that you cannot simplify and under- 
stand them all. But, it would seem that if there were any 
basic information that would serve as a beginning point 
to evaluate this type of program, namely a program claiming 
to offer entry level job training, it would be whether people 
became employed and what their employers thought of their 
training. Now perhaps if there is a low placement record, 
there are good reasons for that and those can be explained. 
Maybe there are particular characteristic^.' of the program that 
make that type of criteria inappropriate ai.'^' that can be 
explained also. Yet, it would seem that you have to start 
somewhere in looking at a program to determine its use- 
fulness; and I think that is why Congress settled on 
starting right there. 

Parenthetically, I would like to deal with a criticism 
I have heard of congressional action with regard to this 
new requirement. Some people say that if Congress wants 
to have such an evaluation made of the uses of federal 
funds for vocational education , it should mandate the same 
requirement for the programs of higher education supported 
with federal funds . Although on its face this criticism 
has some merit , it must be remembered that there is a basic 
difference in the manner in which Congress provides funds 
for vocational education and for higher education. 
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Vocational education really receives general institu- 
tional assistance with certain broad requirements placed 
on its use. Hiahor education does not receive institutional 
support; rather, the vast bulk of federal support for higher 
education goes out as grants to students and then each 
student decides where he or she wants to attend school 
and brings that assistance with him or her to that college 
or university . So, federal support for higher education 
has an inbuilt "quality" standard by relying on the judg- 
ments of millions of students as to wnich institutions 
they believe are the best for them or the most appropriate 
for them. 

Vocational students cannot make that choice since the 
federal funds are qiven to state vocational administrators 
who grant them to loca 1 vocational schools . So, the 
"quality" check in this situation is now meant to be how 
these schools ' programs fare in the newly required evalua- 
tions . 



IMPLEMENTATION 
NI E Report 

Recently, I was qiven a draft copy of a report to the 
National Institute of Education on the status of the 
states* implementation of these new evaluation requirements 
specified in the 1976 Amendments. That report states the 
fol lowing cone lus ions . 

First, it seems that approximately a third of the states 
are using their present systems for evaluation to comply 
with these requirements. . Some of those states say that 
their systems comply, and others say that they do not want 
to change their present systems very much and so they will 
stay with what they have. There seems to be a second group 
of states , another third , which has gone about trying to 
change their present systems in order to bring them more 
in line with the spirit of the law. And finally, there 
is a third of the states which is just starting to implement 
some evaluation features. Some of them have started pilot 
procedures and are movincj on to regular procedures next 
year . 

Consequently, it is apparent that the states have a 
way to go before they will be able to meet these new 
requirements in full. It is understandable that the states 
are in this situation because up to now there have not been 
those types of strict requirements. But the states must 
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continue to press on with their implementation if they 
really care about funding the programs which will provide 
the best possible training for their citizens. 

Furthermore, the results of these evaluations are meant 
to mesh with the other three types of requirements which 
were put into the Vocational Act in 1976. If the evalua- 
tion data obtained are good, they should be put into the 
planning process so people will know what type of programs 
are successful and which programs are not successful. 
It would seem that if one secures good evaluation data , 
that type of data can be used at all different levels of 
decision making. in so doing better planning, greater 
accountability, improved data and good evaluations will 
mesh together, hopefully to bring about better programs 
in vocational education. 



Title I Evaluations 

The last thing I would like to discuss is what has 
happened with Title I evaluations and how that might have 
some meaning for what you are embarking upon right now. 
If I understand a recent report correctly, it seems that 
a strong effort to do evaluation on a comprehensive scale 
is a new thing in vocational education in most states. To 
do evaluations in a systematic manner seems to be very new 
for the vast majority of the states. Therefore, something 
might be learned by looking at what has happened in 
another education program where these types of evaluation 
requirements have been in effect for a much longer period 
of time. 

Title I of the Elementary and Secondary Education 
Act of 1965 is the major federal program of aid to elementary 
and secondary education, involving approximately three 
billion dollars of federal funds for compensatory education 
for poor children. There have been legislatively-required 
evaluations for Title I programs since the mid-60s. 
Title I was the first major social program which included 
requirements that state and local evaluations of the programs 
be conducted. The amount of money spent on evaluations of 
that program has gone from one million dollars to over 
twenty million dollars in the last several years . 

A report that was just done by the Stanford Research 
Institute in Palo Alto on what has happened with the 
evaluation requirements in Title I is directly relevant 
to what you are discussing at this conference. The first 
thing this report points out^ and I know it to be correct. 



-112- 



is that the congressional intent behind the Title I 
evaluation requirements was that local people should 
look at their programs periodically and try to evaluate 
them to see what could be done better and, hopefully, 
to put those improvements into their programs. From the 
beginning, the feeling on the congressional level has been 
that the primary purpose of these requirements is to try 
to bring about local program change. 

This local perspective for the requirements was not 
carried out by the federal administrators of the program. 
Instead, the U. S. Office of Education went in the 
opposite direction in implementing that provision. Almost 
from the first days of the act the U. S. Office of Education 
tried to get nationally uniform data, data which could be 
aggregated at the state level, then aggregated at the 
national level. This would enable them to come tc» Congress 
and the administration and to tell them exactly the number 
of successes or failures in the national program. The 
only way they could see themselves doing that was to more 
or less impose, although sometimes indirectly, the use of 
reading achievement tests. 

Consequently, what one finds in Title I, as you go 
all the way down the line, is that reading achievement 
tests are for the most part, used as the only criterion 
for success in that program. What that has meant is that 
what is beirg required nationally in order to achieve some 
type of national data has resulted in these evaluations 
having almost no influence on the local level in terms of 
changing those programs. In other words, it seems that the 
requirement that started out initially with a focus on the 
local level has been thwarted over the years in an attempt 
to get national data. What has been achieved to date is 
pretty well irrelevant to the people back on the local level. 
It may or may not be relevant at the national level. 

Congresspersons are very happy to hear that the reading 
achievement scores in T_tle 1 have gone up by so many 
months because they can claim that is a wise expenditure 
of money and they like to qo to the appropriations com- 
mittees and tell them they have to put more money into 
the Title I program because the scores have gone up. 
And yet, if that is the sole use of the money, or the sole 
use of the evaluation data, it seems to have frustrated 
the primary purpose of the requirement. 
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You should keep this Title I evaluation phenomenon 
in mind as you try to implement the new evaluation require- 
ments of vocational education. If all of you work for 
several years to secure some type of data which may or may 
not be accurate, but which can be aggregated and can be sent 
on to Congress, then you will not have carried out the 
primary purpose of the evaluation requirements. The primary 
purpose is to try to give local people, and state ad- 
ministrators, an opportunity to learn how their programs 
are operating and to help them improve those programs. 

This local purpose for evaluation may be difficult to 
carry out because, as the report on Title I points out, 
there are several reasons why local people say that Title 
I evaluations are pretty much irrelevant to their decision 
making, even in addition to the nationally-required use of 
achievement tests. These reasons directly impinge upon 
what you are embarking upon. These factors are: 

1. The stability of the program. As with many 
programs , Title I programs live on and on with 

a certain momentum to them. And if that is true 
in Title I, you can bet that it is true with 
vocational education, possibly to a greater 
degree . 

2. The timing of evaluation . It seems that most of 
the evaluations produced in Title I just come at 
the wrong time of the year for them to be of much 
use to people who are reviewing the local programs. 
Again, that is directly relevant to vocational 
education . 

3. Minimal linkage between evaluators and administrators . 
There is very little connection in most school 
districts between the Title I evaluators and the 
Title I administrators . This is especially true 

when outside evaluators are used in Title I. It 
seems that what they produce is not of great relevance 
to the people who really make the decisions in how 
to shape Title I programs . 

4. Many diverse audiences . Title I has many different 
audiences and these audiences have different 
perceptions of what the programs are supposed to 
do. Also , they have different ideas of what type 
of information should be produced in evaluation. 

A number of audiencers feel that achievement test 
scores are not a valid criterion upon which to 
solely judge the program. That problem of course 
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is one vocational educators face very intimately 
in trying to judge what type of audiences are to be 
served, what type of criteria should be used. 

5. The state of the art of evaluation . This is a 
developing field and currently involves many 
divergent evaluation strategies. 

6. Evaluations are perceived at the local level 

as a threat . If an evaluator reports a reading 
score on a Title I program which has not gone up 
to a certain degree, the Title I teacher or 
c-irri.^is trator has a fear of being called in on 
the carpet. Vocational educators face that same 
problem. 

7. Problems are explained away . Regardless of what 
problems are contained in an evaluation report, the 
Title I staff is usually able to "explain away" 
why those things are there and be able to point 

out that one is not really looking at the right 
things and that there are other things to be 
considered. 

Those seven things are more or less things which everybody 
deal''*ng with evaluations of programs is going to face, and 
the task is not easy. 



SUMMARY 

This conference is a beginning of a significant 
discussion of whore we should go with vocational education 
evaluation. Ther*3 are so many variables and so many 
differences of opinion. A lot of things are going to have 
to be discussed and discussed over a long period of time 
so that we can gradually work our way towards a valid 
system of evaluation. In my opinion, the evaluation 
requirements in the Vocational Education Act are probably 
the most significant requirements in that Act , because over 
time they potentially could have the greatest effect on 
programs . 

A number of vocational educators say that they cannot 
terminate local programs even when employment demand and 
placement for certain programs is quite low. For example, 
every high school wants to have a beauty culture course 
and there is no way state or local officials can go to 
those people and say your students are not being trained 
in an area of need. If you want to have that course. 
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you must pay for ic out of your own money. People at 
state and local levels have difficulty doing that now 
partiallv because the data are just not there to show that 
that is so. 

Since we are beginning to face a :;ituation where new 
dollars for education are going to be hard to come by, 
we must have some idea of which programs should be supported 
with our funds. Potentially, these new evaluation require- 
ments in vocational education can give us some facts on 
which to base our decisions. And, as citizens, I am sure 
that we all want the most judicious use of our tax dollars. 
And as citizens, we want to make sure the programs offered 
are going to give students the best type of job training 
opportunities. You have an opportunity at this conference 
to debate those issues and to qive us the beginnings of 
some answers. 
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OUTCOME STANDARDIZATION FOR COMPLIAI^JCE 
OR DIRECTION: THE CRITICAL DISTINCTION * 

Standardized outcome measure s- -ruin or salvation . 
Although the statement is reminiscent of sensational 
journalism, it does serve to anchor the extreme points 
of opinion with respect to the standardization of outcome 
measures in vocational education. Because the issue 
of standardized outcome measures is so salient to voca- 
tional education, I believe that it would serve a useful 
purpose to analyze the need and use of standardized outcome 
measures from the triple perspective of the federal, state 
and local leve Is . 

The impetus for the use of standardized outcome 
measures emanates from federal vocational education 
legislation. Congress, in enacting the Education Amendments 
of 1976 - Title II, stipulated that a vocational education 
reporting and accounting system be established. This data 
system, commonly referred to as VEDS, is to be based on 
uniform definitions and is to contain elements descriptive 
of vocational education students, programs, program completers 
and leavers, staff, facilities and expenditures. Examina- 
tion of House and Senate reports accompanying the legisla- 
tion reveals that VEDS was established to overcome the lack 
of adequate data describing the vocational education 
enterprise in this country. A common complaint echoed in 
the testimony was that vocational education data were not 
compatible across states and hence that aggregations at 
the federal level were of questionable validity as indicators 
of the status of vocational education. This situation was 
frequently described by the adage that "you can't mix 
apples and oranges and get anything but fruit salad. " 
Given the pervasiveness of the problem and the harmony of 
voices raised in protest of the lack of adequate data. 
Congress responded by mandating the establishment of a 
nationally uniform data reporting and accounting system 
in vocational education. 



Presentation by Donald W. Drewes. 
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In addition to VEDS, Congress introduced further 
standardizations by requiring that all state vocational 
education programs purporting to impart entry level job 
skills and receiving assistance under the Act be evaluated 
according to the extent that program completers and leavers 
find employment in occupations related to their training and 
are considered by their employers to be w;ll-trained and 
prepared for employment. This requirement, coupled with 
the requirement that evaluation data be included as part 
of the VEDS , made standardized outcome moasures in voca- 
tional education a reality. Given the existence of this 
reality, I would like to spend some time tracing the 
implications for vocational education. In order to do 
so, it is necessary to examine the effects of standardized 
outcome measures as they impact across federal, state and 
local levels. 



The Federal Perspective 

Melvin Barlow noted in his bicentennial review of 
vocational education that the greatest influence on 
vocational education has been the Congress. This influence 
is currently focused on the use of information in vocational 
education to improve rational data-based planning. The 
philosophy underlying the recently enacted legislation 
is that better data will result in better decisions^ and 
that better decisions will ultimately improve the quality 
of vocational education. Concurrent with their emphasis 
on the need for information to support better decision- 
making is the requirement for information to monitor the 
progress of vocational education to ensure that the intent 
of the legislation is being carried out. It is these two 
themes--the need for information to support improved 
decision-making and the need for information to support 
monitoring of the status of vocational education — and 
their complex interplay that provides the Congressional 
rationale for standardization of outcome measures. 

Congress , in carrying out an expanded monitoring role , 
provided for the flow of standardized outcome data from 
the locals, through the state to the federal level. This 
information is to be provided by VEDS to the Commissioner 
of Education who is required to submit an annual report 
to Congress on the status of vocational education. This 
report is to contain an analysis of the data, presumably 
to determine its policy implications, and is specifically 
mandated to contain a summarization of the outcomes of 
vocational education as measured by the standardized outcome 
data. So as to have an independent check on the status of 
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vocational education, Coi.iuress also provided that the 
National Institute of Education undertake a thorough 
evaluation and study of vocational education at state and 
local levels and report its findings to the president and 
to the Congress no later than September 30, 1980. One 
can but surmise that these data will be used for continuing 
oversight of vocational education. It is quite conceivable 
that Congress will publish an oversight report on the 
implementation of the Educational Amendments of 1976. 
Certainly the increased aval lability of standardized outcome 
data cannot help but influence Congressional opinion as to 
the ability of vocational education to respond to prevailing 
economic and social needs. The valence of this opinion 
will hdve its impact on subsequent Congressional appropria- 
tions and will undoubtedly shape the format of federal 
vocational education legislation. 

The effects of standardized outcome measures will 
extend beyond the halls of Congress. The administration 
position with res[)ect to vocational education could easily 
depend upon the image of vocational education as portrayed 
in the annual status reports. The result of their per- 
ception might well bo reflected in the administrative 
budget with allocations to vocational education conditioned 
by the administration view of vocational education's ability 
to impact on significant social and economic problems of 
immediate political interest. 

Advocacy of vocational education at the national level 
would be facilitated by the availability of standardized 
outcome measures. Professional organizations like the 
American Vocational Association would have access to 
information documenting the progress of vocational education 
and the extent of unmet needs remaining to be served. 
Given the credibility of the process used to generate these 
data, this capability to document progress and needs might 
increase vocational education's competitive position with 
respect to the share of federal funds received. The same 
positive results might accrue to the National Advisory 
Council. Availability of a pool of standardized output 
inf ormat ion should provide them with a data base for 
the monitoring of the progress of vocational education 
in meeting the national manpower needs. 

While offering the possibilMty of positive effects 
at a national level, uniform data on the quality of 
vocat ional education m ight have some unf orseen consequences . 
Provision of a single data base will result in all agencies 
using essentially the same source of information on which 
to base their decisions. The possibility of all actors 
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at the national scene having access to a common data 
base poses some interesting questions. For example, 'Will 
the advantage go to those who have the capability to make 
the most astute analysis of the data?' All of us who have 
experience in data analysis know that data present no 
clear cut evidence as to underlying causal factors. 
The same data lend themselves to multiple interpretations 
with differential policy implications. Given this to be 
the case , one might rightly be concerned as to whether the 
advantage will go to those who are most astute in the use 
of data to buttress arguments that are supportive of their 
position . Since agenci 'js wculd be us ing the same inte 1- 
ligence system, agencies like the National Advisory Council 
might find it difficult to maintain an independent and 
impartial vantage f )oint . Efforts to acquire independent 
data for purposes of verification would be so limited in 
comparison with the size of the national system that its 
utility for verification purposes would be questionable. 

Use and ultimate uti li ty of a standardized vocational 
education data system will depend on whether this system 
is pr imari ly accounting or decision-oriented . An accounting 
orientation wi 11 predispose the collection and reporting 
of standardized indicators of the status of vocational 
education . These indicators would be periodically re leased 
in the form of reports with fixed format and content. 
The purposes would be to provide descriptive information 
on vocational education students, programs, expenditures, 
staff and outcomes as measured by follow-up studies of 
completers and leavers. The principal utility would be to 
provide baseline information on the progress of vocational 
education. Since data elements would theoretically be 
based on uniform definitions and standardized collection 
procedures, data could be aggregated at the national level, 
thereby avoiding the traditional 'apples and oranges' 
problem. Vocational education status indicators would be 
similar in concept to the national labor market indicators 
collected by the Bureau of Labor Statistics and demographic 
data collected and reported by the Bureau of the Census. 

In contrast , the dec is ion-oriented approach would 
emphasize the analysis of standardized data to support 
policy decisions. Rather than providing a static accounting 
of the progress of vocational education, a decision- 
oriented system would be structured so that the data could 
support the identi fi cation , formulation , and choice of 
decision alternatives. The emphasis would be more on the 
use of historical data as a basis for anticipating future 
trends and the analysis of data to support testing of 
hypotheses about the causal factors interacting to produce 
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the observed results. Because of the need for data to 
support decision making, qreater flexibility in output 
format would be re^iuired, v;ith implications for data 
storage and retrieval. The data system would have to be 
structured to allow greater accessibility to the data files 
on an as needed basis. 

The development of a standardized vocational education 
data system will both shape and be shaped by the context 
in which it evolves. The accounting orientation has the 
support of precedent and is congruent with the accountability 
thrust of the present legislation. One would anticipate that 
an accounting-oriented vocational data system would place 
primary emphasis on the production of vocational education 
statistics descriptive of the condi tion of vocational 
education . On the other hand ^ one would anticipate that a 
decision-oriented data system should be capable of providing 
information that enhances the quality of administration of 
vocational education at the federal level . More specifically , 
one might argue that improved access to standardized data 
should have utility in occupational and manpower planning, 
identification of issues and problems of national sig- 
nificance requiring applied research and development, 
sharpened perspectives on forthcoming issues and challenges 
facing vocational education, and improved integration of 
vocational education into a comprehensive manpower delivery 
system. 

The most critical consideration governing the use of 
standardized data at the federal level will be whether 
data are used primari]y to ensure compliance or to support 
improved leadership. A choice for compliance will result 
in the data being used [:'rimarily to monitor the performance 
of vocational education for the purposes of accountability. 
Emphasis on the decision-making orientation will result 
in data being used to support a stronger advocacy position 
for vocational education. 

The use of data at the federal level is constrained by 
the nature of the f edera 1 -state relationship. The consti- 
tutional separation of powers places responsibility for 
education at the state leve 1 . Consequently the federal 
role in the federal-state partnership , as historically 
defined , has taken the shape of federal financial aid to 
the states to assist them in dealing with problems of 
national concern as defined by Congress. The states, in 
principle, are free to establish their goals. Once goals 
are established, it is the federal responsibility to 
determine that means chosen to achieve these goals are in 
accordance with legi s 1 a t i ve i ntent . This partnership 
relationship by its very nature is conducive to use of 
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data as a measure of compliance. As a result of the 
emphasis on accountability r contemporary data systems 
have been primarily accounting type data systems. 



The State Role 

Because the constitutional authority for education 
resides with the states, Congress has assumed that the 
responsibility for vocational education also rests with 
the states. As exemplified in the Educational Amendments 
of 1976, the state is cast in the role of a master planner. 
The state plan as the master blueprint, in accordance with 
the principles of rational planning, is to contain a 
statement of the need for job skills within the state , the 
goals that the state will seek to achieve in satisfying 
these job skills , and a description of the programs and 
courses to be offered to achieve these goals. In addition, 
the plan is to include a description of the use of federal, 
state and local vocational education funds for the achieve- 
ment of these stated purposes. 

The legislation draws no distinction between the state's 
role in goals planning and in its role in program planning. 
Goals planning is concerned with the determination of what 
needs exist to be fulfilled- Program planning, in contrast, 
refers to the way that resources are combined and marshalled 
to serve identified needs. In many instances, states have 
delegated responsibility for program planning to the local 
level. The result is that although the state may be able 
to identify goals, they do not have direct control over the 
delivery system to achieve these goals, the reason being 
that in most instances the responsibility for the ultimate 
delivery of vocational education programs resides at the 
local level. 

Although the states have constitutional authority for 
education, they have in effect delegated this authority to 
the local level- As a result, the state is in much the 
same position with respect to the locals that the federal 
government is in with respect to the states. The effect 
is that state use of vocational education data has evolved 
mainly as an accounting function in response to federal 
reporting requirements. Since relatively few states are 
in a position, nor would they wish to impose the state will 
on the locals, there has been little need for information 
systems to develop to support rational centralized program 
planning. As a corollary, there has been relatively little 
need for state systems to include output measures of local 
performance. Because of state law, state board of education 



policies, and the relationship of vocational education to 
general education, there is relatively little discretionary 
authority over the flow of federal and state funds to 
locals . 

The disparity zween the de_ jure structure of the 
legislation and th- de facto structure of reality is 
exemplified in the difference between state and statewide 
needs. State need connotes that a need is defined by the 
state agency by virtue of its centralized authority. 
Statewide need, in contrast, connotes a need that is 
pervasive throughc :t all the local units comprising the 
state as a geo-political entity. Statewide in this 
context becomes operation My defined as a need experienced 
by a majority of locals. This definition tends to shift 
responsibility for need determination from the state to 
the local level. 



The Local Scene 

Decision making at the local level is frequently 
influenced more by political considerations than the 
need for data to drive rational planning models. Need 
for programs is often based on community rather than 
statewide considerations. Local advisory councils represent 
a widely used means of gathering information about local 
needs and concerns. Other sources of community input 
include school board members who are generally influential 
community citizens, community pressures focused on the 
superintendent and other school administrators, and the 
community linkages maintained by vocational education 
teachers in the conduct of their courses. 

A host of associated factors mitigate against rational 
data-based planning as interpreted by planning scholars. 
Decisions at the local level are frequently constra ned ' y 
scarce resources. Since state monies tend to f]o\% iccoraing 
to enrollment statistics, administrators are reluctant 
to drop courses that are popular and hence paying their 
way simply on the basis of outcome statistics showing 
placement rates in occupations related to training . 
Educational resources are frequently not easily transformed 
into other uses. Staff, equipment and facilities, once 
acquired to support particular program, offerings, are not 
easily shifted to accommodate newly emerging demands. 
The scarcity and frequently limited substitutability of 
resources tend to restrict the discretion of local decision 
makers and thereby reduces the utility of data to con- 
tribute to improved program planning. \ 
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Because the accounting (indicator) data frequently 
required for reporting purposes are of limited use in 
local level decision making, the collection and reporting 
of such data are largely an overhead cost. Data collection 
and reporting efforts are geared to minimize this overhead 
cost with consequent effects on the quality of the data 
reported. Since many of the locals understand the 
economics of this trade-off and its consequences on the 
quality of data reported, they express little or no 
confidence in higher level aggregations of this data. 

Although this discussion pertains to data systems in 
general, the application to standardized outcome elements 
is immediate. Local schools with established vocational 
programs linked into the community infra- structure , are 
not going to be dissuaded to stop offering a program simply 
on the basis of output data showing low placement rates 
in occupations judged by an external standard to be 
related to the program. The program may , for example , be 
a popular program with the student. The instructor may be 
well liked by both students and community and the program 
may be perceived by the community to be serving a useful 
community function. A prime example of this phenomena 
is vocational agriculture . By a strict interpretation of 
production agriculture, placement rates might be low for 
a particular program, yet the program may have widespread 
community support in rural communities. 

Statistical data that are inconsistent with the nexus 
of information regarding program support will tend to be 
neutralized by challenging the validity of the data. 
This challenge can take place on a number of grounds. 
A likely basis of challenge would be on the definitions 
and values underlying the data. With respect to standardized 
outcome measures , this challenge would focus on the validity 
and acceptability of the concepts underlying the data 
source. Criticism of placement rates would likely take the 
form of criticism of the definition of related occupations 
and a challenge as to the efficacy of placements as the 
sole indicator of the value of vocational education. 

Again , re ferring to the vocational agriculture example , 
the validity of the data might be challenged on the 
grounds that production agriculture represents but a 
small fraction of the occupations that require the skills 
developed in vocational agriculture programs. This 
argument might also be augmented by the contention that 
vocational agriculture teaches a philosophy and a way of 
life that is urgent for the preservation of the values 
and morality of rural America. The negative effects of 



low placement rates might be further countered by the 
argument that vocational education prepares people for 
work rather than for specific jobs. Using this reasoning, 
the data could be said to provide misleading results in 
that they did not depict the number of vocational students 
who created their own employment because of skills learned 
in vocational programs, nor did they account for the 
contributions made by vocational education to the subsequent 
career achievements of vocational students. 

The purpose of these examples is to illustrate that 
the use of outcome data is largely a matter of attitude. 
If the data tend to be in accord with prevailing attitudes, 
the results will tend to be accepted as a rationalization 
of the validity of the foundation values and beliefs. 
If/ on the other hand, the data are at cross purposes v/ith 
these fundamental values and beliefs, the validity of the 
data will be challenged in an effort to reduce the dissonance 
created by the incongruent piece of evidence. Those of you 
acquainted with psychology will recognize this as an example 
of cognitive dissonance theory which states that a person 
will act to reduce the dissonance created when information 
conflicts with values. Basic values and beliefs change 
slowly and only after data have repeatedly shown that these 
beliefs and values are inconsistent with reality or lead 
to actions with adverse consequences. 



State -Local Relations 

States are often reluctant to take punitive action 
on what may be interpreted as adverse outcome data. 
For one thing, many state vocational educators share the 
same apprehension and reservations about the ability of 
outcome data to capture the essence of quality. They are 
also astute enough to realize that neither the state nor 
the federal government can mandate quality programs. 
Since many of the state supervisors and consultants have 
carefully nurtured a network of relationships with local 
personnel, they are unwilling to jeopardize the continuance 
of these relationships by taking precipitous action on 
the basis of information contained in standardized outcome 
data. They realize that whereas they may be able to force 
a small change in program design and operation, a show of 
force would ultimately reduce their effectiveness as 
facilitators of long-run program improvement. State 
level vocational education professionals also in the 
main realize that positive change is a slow evolutionary 
process. State input, if it is to effect change at the 
local level, must first be accepted at the local level. 
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This acceptance depends upon the extent to which the 
information is regarded as credible and the source, 
legitimate. The process of establishing credibility at 
the local level is slow to build since it depends upon 
a relationship of mutual trust. The real business of 
facilitating programs is done on a personal basis through 
informal relationships. Thus, the formal data flow is often 
a matter of formality with decisions having been reached 
by common concensus. in this personal relationship, 
the state partners generally respect the sovereignty 
of the local to be in the best position to know its local 
needs and conditions . 



The formal- in formal relationships between state and 
local are paralleled by the flow of information. Formal 
information tends to be that required as a matter of 
compliance and is generally imposed by federal and state 
laws and administrative policy. Whereas the formal 
information structure supplies the data for reporting 
requirements, the informal information structure generates 
much of the input into the decision-making system. As 
a result, the formal structure acts as a pipeline or conduit 
of information for data reporting purposes that flows 
upward through the state and ultimately into the sea of 
federal information. Because this flow tends to be 
isolated from the informal system, this information has 
relatively less impact on decision making and serves mainly 
for satisfying compliance purposes. The state in this 
process functions much as a centralized warehouse, collecting 
information from the locals, packaging it, and forwarding 
it on to the federal destination. 



Whatever problems existed at the state level with 
respect to mandating quality are magnified manyfold at the 
federal level. The federal level, being further removed 
from where the action is, has to rely on the data generated 
by the formal system for information as to the current 
status of vocational education. Lack of informal information 
makes the federal level dependent upon formal information 
for intelligence with the result that the data, including 
standardized outcome data, present a rather cloudy, 
incomplete, and frequently inaccurate picture of voca- 
tional education. 



Suggestions for Improved Utilization 

Given the structure of vocational education and the 
impact that this structure has on information, the question 
of immediate concern is what can be done , if anything , to 



increase the effective utilization of information by 
vocational education decision makers. To say that I have 
the answers to these questions would be presumptuous, I 
can only share with you some views and notions as to how 
the process might be modified with the hope of improved 
data utilization. 

I believe it imperative that we distinguish between the 
use of data for reporting purposes and the use of data for 
program improvement. In recognition of this distinction, 
I propose that we find a small core of standardized data 
elements that will serve to indicate the status of vocational 
education. The data elements to be collected should be 
judiciously chosen so as to present the maximal amount of 
information to the major users of vocational education data 
at the national level. The dimensions of this core of 
data elements should span the six categories provided for 
in the legislation. Specific elements within each of the 
six categories could be chosen so as to answer the most 
frequently asked questions of vocational education. Since 
Congress is the major consumer of vocational education 
information at the national level, it would seem appropriate 
to use their informational needs as criteria in selecting 
the data element composition for the core of indicator 
items . 

The temptation to expand the core of indicator variables 
collected should be resisted, lest the burden of collecting 
this information on a regular basis become too burdensome 
and costly for the benefit accrued. Because no data core 
regardless of its size can answer all the questions that 
might be raised, provision should be made for rapid survey 
capability to secure answers to questions, providing that 
there is sufficient need. One salient indicator of need 
would be the willingness of Congress or a federal agency to 
appropriate money for the conduct of the survey. 

To insure comparabi li ty of the indicator measures , 
their format should be rigorously controlled. This could 
be achieved by uniform definitions and standardization of 
the collection procedures. Development of uniform 
definitions and standardized collection procedures should 
be under the auspices of the agency having responsibility 
for the collection of indicator data. Since the National 
Center for Education Statistics has legislative responsi- 
bility for the education data function, this agency would 
be a logical choice. The elements should be operationally 
defined and the collection procedures based on a sound 
technical and statistical methodology. 
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The data elements should be structured so as to permit 
rapid retrieval of information in a flexible format as 
defined by the user. The flexible format would circumvent 
the present difficulty of information being available only 
in the tabular report forms chosen by the report writers and 
updated only as often as the agency publication schedule 
permits . Those of you f ami liar wi th federal report 
publications know that this updating often entails a 
considerable time lag, 

The role of the state and locals in this collection 
process varies considerably. One option would be to use 
Arthur Lee's id^a and have the locals submit data on 
individual student records in machine readable form. 
This would circumvent the rather cumbersome current process 
and would e liminate the laborious activities associated 
with f i lling out current reporting forms . However , this 
assumes that most local schools have data process ing equip-- 
ment. Another possible limitation is that the sheer 
nuinber of records involved would exceed the processing 
capacity of a centralized syrtem. 

An alternative to securing information directly from 
the locals would be to use the state system as an inter- 
mediary. State systems could be used to preprocess the 
information obtained from the locals and could send the 
processed data to the national leve 1 in machine readable 
form. Since most states have automated data processing 
capabilities of some sort, this would overcome t-.he equip- 
ment limitations associated with having the locals directly 
involved. The states might also exercise some preliminary 
processing such as reliability and validity checks on the 
data to ensure that they are in proper form* By decentralizing 
some of the responsibility to the state level, the states 
could be more involved in the process and the burden 
imposed on the central processing unit could be reduced. 
In this sense, the system could function much as a 
distributed computer network . 

So far I have not touched on how the reporting burden 
may be reduced . This could be accomplished in a variety 
of ways. First, the reporting burden could be reduced 
by restricting the size of the common core of indicator 
variables requested. Locals might be paid to provide the 
information according to federal reporting standards . 
This would make for a more equitable relationship since if 
the federal government were paying for the d^xta they certainly 
could expect to have it reported according to their specifica- 
tions. Another option would be to pay an independent 
agency to sample data from the schools much like the 
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Census Bureau now secures current population data from 
samples of households. It is conceivable that the state 
agency might be paid a stipulated amount to secure this 
information on a specified sample of units according to 
the standardized procedures. 

The major question raised by the critics would be 
that of cost. I maintain that the cost would be no 
greater and might result in a savings. Indicator data 
collected on a well-drawn sample could provide data of 
sufficient precision for federal purposes at significantly 
less cost than the universal sampling currently being used. 
The fact that schools would be paid for thei r effort 
expended in collection or data according to externally 
de fined standards would certainly do much to improve the 
quality of the data collected. The major difference in 
this proposed method and that currently being used is that 
the cost of collecting data would be directly borne by 
the federal government rather than indirectly shifted to 
the states and locals . Funding for this data collection 
effort could be achieved by several methods. Congress 
might provide a special appropriation to NCES for performing 
this function. Another option would be to transfer a 
stipulated amount of vocational education appropriations 
to NCES, or a third option might be to stipulate that the 
states use funds authorized for planning under Section 
102 (d) for purposes of data reporting. 

The effect of these recommendations would be to test the 
utility of data indicating the status of vocational 
education. If these data have utility at the national level, 
then Congress should be wi.lling to pay for the collection 
of these data in the same way that it provides for the 
collection of unemployment and employment data and census 
indicators. If there are no advocates for the collection 
of data at the national level, then it is not cost effective 
to collect and should be discontinued. 

To say that data only have use for reporting purposes 
would be a pessimistic prognosis for the future of vocational 
educat ion . I am opt i mis tic that the qual i ty of vocational 
education decision making can be perfected by the provision 
of improved data . Furthermore , I believe that the route 
to significant improvement is through the medium of tech- 
nical assistance. The focus on assistance is consistent 
v/ith the purpose of the 1976 Amendments and provides a 
more positive approach to improvement than that provided 
by a compliance emphasis. I believe that if the federal- 
state-local partnership is to be more than rhetoric, then 
each [partner must bring something to that partnership. 
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The federal and state contribution to that partnership 
is assistance provided in a spirit and form to augment 
and strengthen local operations. The local role in 
this partnership is to accept assistance in the spirit 
in which it is offered and to rise to the challenge of 
a quest for continued improvement. 

A national data base of standardized indicator variables 
could provide the mechanism to support the provision of 
technical assistance at the federal level. Three principal 
agencies could be cooperatively involved in the assistance 
effort. The Bureau of Occupational and Adult Education 
(BOAE) has legislative authority for the administration 
of vocational education legislation. The National Center 
of Education Statistics (NCES) has legislative authority 
for the development and operation of a national vocational 
education data system. The National Occupational Informa- 
tion Coordinating Committee (NOICC) has responsibility 
for the design and implementation of an occupational 
information system to serve the national, state and local 
needs for occupational related information. Working 
in close conjunction, these agencies could exert a 
powerful force for the improvement of vocational educa- 
tion through the increased utilization of standardized 
data. Analysis of the national indicator data to de- 
termine the implications for vocational education would 
provide an informational basis for the provision of 
technical assistance. 

Capability within the Bureau of Adult and Occupational 
Education to analyze data for its policy and management 
implications would contribute significantly to the enrich- 
ment of BOAE leadership functions. Exemplary data 
capabilities might be the identification and interpreta- 
tion of emerging trends in the demand for and the support 
of vocational education. Enrollment data could be 
analyzed for shifting patterns and the factors associated 
with these patterns identified. Similar analyses could 
be conducted v/ith respect to vocational education completers 
and leavers with a view toward identifying the individual 
and programmatic factors that contributed to successful 
vocational education outcomes as determined by a variety 
of criteria. Insights into the factors associated with 
vocational education performance as measured by the stan- 
dardized outcome indicators would serve as an objective 
basis for identification of problems of national signif- 
icance. Such data offered to the states in the spirit 
of assistance could constitute a salient force for change. 
By providing the state assistance in the interpretation 
o^ the data and its significance for vocational education. 



state and local levels would see some possible benefits 
accruing from the data that they had provided. 

The technical assistance role also includes NCES. 
A principal role for NCES would be to play the lead 
agency role in the development, operation, and updating 
of a policy-relevant vocational education data system. 
Policy relevance would at the minimum, require analytic 
capability in the system to support the determination 
of functional relationships between data elements. 
This would require data-based management and support of 
statistical analysis procedures. Policy assistance to 
other agencies might take the form of simplified computing 
routines and/or the provision of personnel assistance in 
analysis and interpretation. Technical assistance could 
be provided to the states in order to facilitate their use 
of data to support administrative decision making. This 
assistance might take the form of suggested analytical 
procedures that the states could use in analyzing their 
state level data bases, alternative methodologies that 
the state might wish to employ to augm.ent the indicator 
data currently being collected and reported nationally, 
and training packages and conferences designed to 
facilitate the understanding and use of reported data. 
Methodological assistance in the collection and analysis 
of data at the state level would serve to stimulate the 
demand for more and improved data. The important con- 
sideration is that the demand would emanate from the 
users' need for data to support internal operations rather 
than to satisfy external reporting requirements. 

The occupational information system to be developed 
by the National Occupational Information Coordinating 
Committee (NOICC) is yet another potential mechanism 
for stimulating the use of data in response to internal 
needs. Whereas NOICC is to provide the design standards 
for the development of an occupational information system, 
the responsibility for implementing this system at the 
state level rests with the State Occupational Information 
Coordinating Committees ( SOI CC) . The re lationship between 
the NOICC and the network of SOICCs provides yet another 
mechanism for technical assistance. The NOICC, in ad- 
dition to providing standards for the design of an 
occupational information system, could assist the states 
in interfacing and drawing together the large complex 
data producing systems in the state into an integrated 
informational network. Occupational demand data could 
be more effectively linked with supply, and a more in- 
tegrated picture of supply could be provided across 
educational 1 eve Is and agencies concerned with human 
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resource development. The requirement for standardiza- 
tion of definitions across systems could provide a vehicle 
for the linkage of these systems at the state level into 
a more cohesive and comprehensive informational system. 
By linking the existing data systems in the state into a 
confederation of systems based on shared definitions, it 
should be possible to shape the information into a variety 
of formats to suit the needs of various user groups. By 
serving as a vehicle to promote dialogue between various 
user groups and the producers of data, the SOICCs can 
perform a useful function in facilitating improved under- 
standing of data that is currently available, the methodology 
underlying its collection, and the potential uses to which 
this data might be put, NOICC could fulfill a useful 
function by providing technical guidelines and assistance 
to the SOICCs in organizing and focusing state efforts 
on the implementation of a unified occupational information 
system. This assistance could take the form of technical 
nanuals designed to f aci litate state understanding of 
the procedures underlying the collection of certain data 
elements , assistance in performing needs assessments to 
determine the informational needs of various user groups 
at state and local levels, provision of information 
descriptive of the alternative methodology for collecting 
information identified as being needed, and stimulation of 
state interest by providing financial support for the 
collection of information that might satisfy a need common 
to a variety of users. One example of such an area might 
be that of a statewide survey to determine the universe of 
need for education and manpower programs. NOICC could 
promote this activity by providing funds and technical 
guidelines to the SOICCs to assist them in coordinating 
this activity within their respective states. 

State occupational information coordinating committees 
could become the nucleus for the promotion of the use of 
data for the improvement of education and training services. 
For one thing, the explicit purpose of the SOICCs is to 
improve the dialogue and cooperation existent between 
agencies involved in the delivery of education and manpower 
services . A concern for occupational information is the 
common basis and provides the rationale for SOICC organiza- 
tion. Since SOICC exists as a vehicle to focus coordinative 
attention on the need for and the use of occupational 
information as a means of securing improved program planning, 
each state for the first time has a staff committed to 
the promotion of the development and use of a coordinative 
data base. Thus, the SOICCs have the pot-^ntial to launch 
a movement to more actively involve a wide constituency 
of data users and to develop support for increased 
involvement in the data production process. 
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In order for this movement to achieve its fullest 
potential it is imperative that the concept of coopera- 
tion inherent in the SOICC rtructure be extended to the 
local level. The extension of a state level network of 
information users and producers can be extended into the 
regions by the establishment of a network of regional 
information centers-^one center to be established for 
each region. The major purpose of the regional information 
centers would be to function as the hub of a regional 
information system. The major purpose of a regional 
information system would be to interject the uniqueness 
of local labor market conditions into a comprehensive 
statewide information system. Functions of the regional 
information system would be both to collect and provide 
occupational career and educational and manpower informa- 
tion pertinent to the region served by the center. 
Regional information centers might provide information on 
current and projected occupational employment for that 
region, current job vacancies in the region, demographic 
information pertinent to the region, information on current 
and anticipated economic conditions, information on the 
availability of training and education opportunities of 
the region and evaluative data gained from follow-up 
and placement of the clients of CETA and vocational 
education programs as well as other programs that might 
eventually carry out follow-up activity. 

Regional information centers might also be repositories 
for information (from a variety of sources) that pertains 
to the social and economic characteristics of the region. 
With regard to career information functions, regional 
information centers could provide information and referral 
services to people concerned about the availability of 
education and training opportunities in the region. The 
centers could also provide information on available 
assistance in the region for job placement, counseling and 
guidance services or for other programs designed to 
prepare and assist people in finding suitable employment. 
Additionally, they could provide information about duties, 
requirements, wages, and employment prospects for a variety 
of occupations to be found in the region. 

These regional information centers could also supply 
information to a host of local community organizations. 
Examples of organizations receiving input from regional 
information centers might be occupational counseling 
and guidance centers at both secondary and postsecondary 
institutions, local and industrial development commissions, 
planning officers of educational and CETA agencies, 
local community education and work councils, community 
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action agencies, county and metropolitan planning officers 
and a variety of citizen action groups. Data collection 
activities of regional information centers might include: 
collection of information from employers as to job 
vacancies, characteristics of workers customarily hired, 
including skills necessary to perform the job, required 
personal qualifications, training opportunities, and hiring 
requirements. Because of the involvement of local 
representatives in the operation of the center, the likeli- 
hood of employer participation and provision of information 
would be greatly enhanced due to increased rate of return 
for the employer — both in terms of better trained employees 
and the public relations accruing from participating in 
a community-based activity . 

A regional information system could also contain 
information that could be used to assist in the planning 
of vocational education and manpower programs to serve the 
region. Program planning information might include an 
inventory of education and training opportunities in the 
region provided by vocational/manpower program delivery 
systems and training programs provided by private employers. 
Regional information centers may provide a technical 
assistance function through the provision of a wide variety 
of information. This information might include legisla- 
tion--both federal and state — pertaining to the development 
of resources at state and local levels, federal and state 
rules and regulations that might have an impact on local 
planning of education and manpower services and reports 
and other documented results of research and development 
efforts of other states and communities in dealing with 
the problems of developing human resources. 

These regional information centers could serve much 
as public libraries with information available upon 
request. Information from the local vocational education 
data system maintained by CETA prime sponsors, and data 
systems maintained by local employment security agencies 
could be provided to the regional information centers. 
This would make information readily available that would 
provide public knowledge of the effectiveness of education 
and manpower service delivery programs. 

Thus, I have come full circle. Whereas the locals 
are the originators of data describing the outcomes 
of vocational education, they must also be the final 
users of this information if any benefit from them is 
to result in the ultimate improvement of vocational 
education. The proposed system provides this feedback 
link under the general rubric of technical assistance — 
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flowing from the federal to the state and finally through 
the regions to its ultimate use in the decisions that 
shape the process and ultimately the final outcomes. 

The challenges are great . However , I believe that the 
time for action is now. Legislation has created a mandate 
for the improved use of information and has created a 
variety of mechanisms to support this improved use. 
Whether history will ultimately record these me ch an isms 
as yet another futile attempt, or whether they will 
provide the means to move us to a new plateau depends 
upon our foresight, ingenuity and imagination in structuring 
these mechanisms to serve our needs. I am hopeful that 
vocational education will play a lead role in these 
pioneering efforts. A 1 though the challenges are great , 
I believe that vocational education has the vitality, the 
ingenuity, and the creativity to overcome these obstacles 
and to move to greater achievements in the promotion of 
human well-being. 



Id'IAC'IMoU (M iin-rriihr: :r]'ANDAKDI ZATION 
FOR COMPLIANCE OR DIRECTION: 
THE CRITICAL DISTINCTION* 

Donald Drewes has, I think, performed a real service 
in focusing attention on two purposes for which standardized 
outcome data--and a national vocational education data 
system--may be used. They are not incompatible, certainly 
not mutually exclusive, and I don't think Don is suggesting 
that they are. What he does say is that the ultimate 
utility of a national information system will depend on 
whether it is primarily accounting oriented or decision 
oriented . 

He makes an excellent case for the decision oriented 
system with its greater flexibility and analytical 
capability as opposed to an accounting oriented system of 
limited status indicators. In the latter we would have the 
collection and reporting of fixed data elements — like 
percentages of the total vocational education enrollment 
represented by the disadvantaged, handicapped, and ethnic 
minorities. In a decision -oriented system, we would have 
the data from which these percentages are obtained, 
collected and made available in such a way as to know 
where the disadvantaged, handicapped, and ethnic minorities 
are benefiting most from vocational education--and where 
they are benefiting least--in terms of occupational programs, 
program levels, kinds of institutions, and even the 
neighborhoods in which they live and the particular 
schools they attend. 

In othei words , Don is saying that the national voca- 
tional education data system, which Congress intends to 
he the basis for planning and program improvement, should 
have enough analytical capability to enable administra- 
tors and decision makers at every level to know the 
particular circumstances under which one or another of 
these target populations is likely to benefit most, and 
which circumstances should be avoided. I believe this is 
what Don is referring to when he says that the most critical 
consideration governing the use of data at the federal 
level will be whether the data are used primarily to 
ensure compliance or to support improved leadership. 



Reaction by Arthur Lee to Donald Drewes' presentation. 
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What Don also seems to be saying is that the decision 
to use th^ data to ensure compliance or to improve leader- 
ship cannot be made after the system has been developed. 
It has to be made now. Actually, the question is really 
whether to develop a system capable of providing both 
kinds of data or only one . We can have compl iance 
accounting from a system that If. also capable of providinq 
data for planning and decision making , but we cannot have 
decisior oriented data from a system developed only to 
produce accounting data. 

One of the problems in deve loping this kind of a 
system, as Don points out, is that most of the state 
systems were not designed for both purposes. Their major 
purpose was, and still is in most cases, to perform an 
accounting function "in response to federal reporting 
requirements." It is a very interesting point that Don 
makes when he says that the state's role is usually not 
program planning at all, but goal planning. Program 
planning is a function of the local school districts. 

What, then, are we doing with all of the emphasise- 
in federal legislation, at least — on program planning 
at the state level? Don suggests that casting the state 
in the role of a master planner results from a mistaken 
assumption by Congress that the responsibility for 
vocational education rests with the states, whereas only 
the authority resides with the states. Responsibility 
has been delegated to the local education agencies. Many 
states simply recognize this by building their state plans 
out of the total of their local plans. 

Don also discusses the difficulty that local ."schools, 
especially, have in trying to change program as a result 
of negative employment demand data. The whole question 
of relating planning and program decisions to employment 
supply and demand has encountered a great deal of op- 
position. We have given it lip service because it sounds 
like something we ougri : to be doing, but since the 
Amendments of 1976 wer'^ passed, I think many vocational 
educators are beginning to object to the whole idea. Don 
mentioned the vocational agriculture people as an example 
of rejecting this kind of data if they were in conflict 
with one ' ^; basic beliefs. Mary Kievit, yesterday, discussed 
the influence traditional values have on policy and change. 
And last night there was quite a discussion between Jack 
Jennings and some of the conference participants about 
using placement data as the basis for evaluating vocational 
education . 
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I wunder too if the concept of employment market 
supply and demand data should continue to be the principal 
measure of vocational education output. Don doesn't say 
this — and I am digressing from his paper for a moment — 
but I wonder if our insistence on that one measure as 
almost the only basis for judqing program success, and 
therefore the major consideration in program planning, 
is one of the reasons that - rat ioaal education data systems 
tend to be accounting; oriented Instead of decision oriented. 
T think we should take Jack Jennings' suggestion seriously 
when he said last night that perhaps Congress should 
reconsider its use of placement as the measure of ac- 
countability in vocational education. 

Before 1963, I don't think there was any question 
about the federal support role being based on this 
concept, and probably most of the states' responsibilities 
as well. I am not sure about the local education agencies. 
Frankly, I doubt if placement was ever as important to the 
local education agencies as student demand and student 
satisfaction. And I think it is still that way. Moreover, 
the federal legislation in 1963 — with the amendments of 
1968, 1972, and 1976 — support that position even while 
insisting on placement as virtually the only measure of 
success. The purpose of that legislation is spelled cut 
in the first tr.ect ion--repeated with every set of amendments; 
it is that, "... persons of all ages and all com- 
munities . . . will have ready access to vocational 
training or retravning which is of high quality . . . 
which is suited to their needs, interests, and ability to 
benefit from such training." 

It was pretty well established in the 1963 Act that 
a major shift was intended from emphasizing programs to 
emphasizing persons served. This was further indicated 
by the emphasis on helping particular groups of persons-- 
the disadvantaged, the handicapped, ethnic minorities, 
emales, and pos tsecondary students. 

It seems to me that the federal purpose in supporting 
vocational education is not so much to place students in 
occupations for which em.fjloyment market projections 
indicate the demand will be greater :, but to ensure 
that every perr ^-n who can benefit from quality voca- 
tional education be cuven the opportunity to do so. 
There is a difference. r.et us go back to Don's example 
of the vocational agriculture advocates who criticize 
employment market supply and demand data as inadequate 
because those data do not measure all the benefits that 
vocational agriculture students may be receiving. The 
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federal purpose says nothing about student placement, but 
does say that all persons who can benefit should have 
access to a quality program. It may be inferred that this 
means a program of their own choice. And this is exactly 
what the vocational agriculture people say they are doing. 

What would be wrong with developing outcome data which 
measure the achievement of the stated federal purpose? 
My own view is that we should continue to measure place- 
ment and placement effectiveness, but that evaluation of 
vocational education-- the basis upon which programs are 
judged --should be shifted from placement alone to placement 
as only one measure of competency .. I think Congress 
should look at other measures of competency, and perhaps 
build them into the next federal legislation. Competency 
testing has been neglected in vocational education, and 
that is one of the reasons for periodic attacks by 
critics. Too often vocational educators who know that 
their students have developed definite levels of competence 
in occupational skills have no way of proving it except 
through the uncertain chance that jobs requiring these 
skills will be immediately available and that all of their 
students are ready to accept them. 

Competency testing as an outcome measure should not 
be too difficult, even with national standardization. 
We have it in apprenticeship programs. We have it in the 
healt^ occupations, we have it in the aircrr.ft maintenance 
field, and if competency were to be measured in specific 
skills rather than by whole occupational programs — which 
often do not match employment demand categories anyway-- 
the whole thorny problem of related occupations raised by 
relying on placement data would be solved. In any case, 
would it not be bettor to know the extent to which persons 
in all communities have access to vocational education 
programs, and t!ie extent to which those programs are of 
high quality as measured by the competence of students 
who complete them, and alpo the extent to which persons who 
need and can benefit from vocational education are enrolled 
in such programs, than simply to know how many are 
placed each year? 

But back to Don's paper, he makes a number of specific 
r-. commendations which I think are excellent. One in 
particular is that only a small core of standardized 
data elements should be collected. C. 0. Tower and I 
developed a core of this kind in one of the Baseline 
supplemental reports in 1974, "A Data Base for Vocational 
Educati:Dn and Manpower Training." Ours contained thirty- 
nine data elements covering students, programs, expenditures, 
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professional personnel, and f inance--each linked with 
ail the others for maximum flexibility and analysis. 
We would probably have to modify it slightly today, but 
not much. The concept which Don is advocating is not 
difficult to implement with automated equipment, and 
would contain essentially the same data elements no matter 
who designed it. 

My final comment is in response to Don's suggestion 
that the national vocationni education data system should 
be paid for by the federal acvernment if it is useful 
enough to justify the cost. I think this is what Congress 
intended in the 1976 legislation, and also in the NCES 
authorizing legislation of 1974. I agree with Don, that 
the federal government should not impose a data collection 
system on the states and the local education agencies 
simply for the purpose of determining whether the states 
and the local education acjencies are in compliance with 
federal policy. Accountability is important, even critical, 
when it involves ijublic funds. But 1 also strongly agree 
with Don that we should have a nationally standardized 
data system for vocational education to be used for 
decision makinci, i)lanninu, and program improvement-- 
and not for accountability alone. 



JOB SATISFACTION AND PERFORMANCE 
MEASURES: THE STATE-OF-THE-ART* 

The primary purposes of th is presentation are to 
(1) analyze the concepts of job satisfaction and job 
performance, (2) develop a model of the causes of each, 
and (3) discuss the possible effects of vocational 
education on each concept, using these causal models. 
Along the way, several important issues are discussed 
which should be taken into account in designing an evalua- 
tion process for a vocational education program. 



Job Satisfaction 

The most common definition of the concept of job 
satisfaction is as follows: 

The affective orientation toward or 
emotional reaction to the j ob ( or various 
components of the job) re suiting from the 
appraisal or evaluation of one's job or job 
experience (Lawler 1973; Locke 1976). 

This definition emphasizes two interrelated points. 
First, job satisfaction is a feeling or an emotion. 
As such, it is likely to have a strong impact on the 
lives of employees. Although job satisfaction is an 
emotion, it is !:iot without foundation. The second 
part of the definition suggests that the feeling of 
satisfaction derives from a (more or less) careful 
evaluation of one's job — what it has to offer in various 
ways. Stated in a different way, there is a descriptive 
part of one's attitude toward the job (What is the job 
like? What does it offer?) and also an evaluative 
component (How satisfied arc you with what the job offers?) . 

Job satis fact ion can re f er to the overall satis- 
faction with the job as a whole, or to satisfaction 
with specific components (or facets or elements) of the 
job, such as [^ay, promotion opportunities, supervision, 
and so forth. These components can be fairly general 
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(e.g., the nature of the work itself) or much more 
specific (e.g., degree of autonomy allowed in dccidinq 
what procedures to use when doing a specific task on 
the job) . 

A list of the most commonly used components of the job 
would probablv include most of the following, although the 
list is illustrative and not exhaustive (summarized from 
Campbell and Pritchard 1976; Lawler 1973; Locke 1976): 
work itself (intrinsic interest, variety, opportunity 
for learning, difficulty, amount , chances for success , 
control or autonomy of pace and methods, responsibility, 
use of abilities, feeling of achievement, opportunity to 
stay busy, authority, chance to do things for others, 
status or prestige); pay (amount, fairness or equity, 
method of payment, possibility of future raises); 
promotion (opportunities for promotion , basis of 
promotion , fairness of promotion, type of promotion 
offered) ; recognition (praise from supervisor, co- 
workers, outsiders); workin g c onditions (house, rest 
pauses , equipment , temperature , vent i la t ion , humidity , 
location , physical layout) ; supervision (overall style , 
supervisor* s influence over others in work place , technical 
skills, human relations skills, administrative skills, 
consistency), co-workers (competence, friendliness, 
helpfulness); company policies (concern for employees, 
specific policies toward unions, retirement, transfers, 
reputation of company, job security). 



A Model of the Causes of Job Satisfaction 

Satisfaction with Job Components . Lawler (1973) 
has summarized and evaluated the various models of the 
causes of job satisfaction, concluding with a consolidated 
model which has gained wide acceptance. A somewhat 
simplified version of this model is presented in Figure 1 
and discussed in some detail below. 

Earlier it was suggested that the amount of job 
satisfaction results from an evaluation of the job. 
Pursuing this point , there is general agreement that level 
of satisfaction (overall or with a specific component) 
is a function of the discrepancy between what the person 
feels should be and what is seen as existing. Using pay 
as an example, if an employee feels she/he should be 
compensated more than she/he sees herself /himself as 
receiving, then she/he will be dissatisfied with pay. 
If the "should be" and "is" are in balance, the employee 
will be satisfied. If the amount received is greater than 



-144- 



Figure 1 



Lawler's Model of the Determinants of Satisfaction with Outcomes 
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"is" received 



Lawler, E. E. Motivation in work organizations . Belmont, California: 
Wadsworth Publishing, 1973. 



the employee believes she/he should get, a feeling of 
"guilt" or "overcompensation" results. (The discrepancy 
in this direction probably has to be fairly large for a 
significant feeling of guilt to occur, particularly for 
an aspect of the work like F^ay.) This part of a model of 
job satisfaction is substantiated by the empirical 
evidence (e.g. , V7anous and Lawler 1972) . 

Although this model is fairly straightforward, it has 
some interesting implications. For example, two employees 
may have, objectively, the same opportunity for promotion 
and yet differ in satisfaction with promotion opportunity. 
The differences in satisfaction may be due to (1) different 
perceptions of the op.po: ^ unity for promotion (the 
"perceived amouiif ^.a.l the model), (2) different 
standards concern-p ; y.^-^t chances for promotion 

"should be," or (::•) .'.r^t of these reasons. This example 
emphasizes the poi. •. ^ the "should be" and "is" part 

of the model are porCi ^'t' ^ ons and are not necessarily 
objective or accurat*'. 

This model of jot^ j-Ij r isf action also includes the deter- 
minants of the "shou.''.c^ Lc ' and "is received" perceptions. 
One of the determina i^. r c i: "should be" are perceived 
personal job inputs. These include all of those things 
that individuals fet.l v^^ey contribute to the job, such 
as skill, experience, eirort, performance, education, and 
so forth (see Figure 1). The higher these perceived 
inputs, the greater the amount of the work outcome of 
reward individuals feel they should receive. For 
example, if an employee feels she/he performs better 
than a co-worker, then perceived inputs are high and the 
amount of outcomes that should be received is affected. 

The model also suggests that perceived job character- 
istics affect the perceived amount that should be 
received. The more demanding the job (e.g., the higher 
the level, the more difficult the work, the more re- 
sponsibility required), the greater the outcomes (e.g., 
pay, intrinsic satisfaction, working conditions) that 
should be received from the job. For example, if an 
employee is given new, demanding job duties, the employee 
is likely to feel that the pay should be greater. 

The determinants of perceived amount of the outcome 
that is received are simpler (see Figure 1) . Actual 
outcomes, of course, have the greatest impact on this 
perception. In addition, however, the amount of the 
outcome believed to be received by referent others is 
important. Take as an example two computer programmers, 
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each earning $10,000 a year. If programmer A's co- 
workers earn $9,000, then she/he will feel her/his per- 
ceived amourit of pay is relatively high. Programmer B, 
however, works alongside other [programmers who make 
$11,000. B will perceive her/his current level of pay 
to be lower. Thus, assuming all other factors in the model 
to be equal for A and B, programmer A will be more 
satisfied with pay than B. 

Before examining othc-r m.odels of satisfaction and 
applying the models to vocational education, it should be 
noted that this model has received a fair amount of 
empirical support. The findiiigs of many older studies 
can be reinterpreted in tnis framework and are generally 
consistent with the theory. In addition, a recent study 
by Dyer and Theriauic (1976) specifically tested most of 
their model and found cieneral support. 

Overall Job S atis fact i on . It is often hypothesized 
that overall joL satisfaction is a result of some combina- 
tion of satisfaction with the various components of the 
job. Further, it seom.s reasonable that the components most 
important to the individual are somehow weighted more than 
unimportant components in determining overall satis- 
faction. For example, ar. employee who values challenging 
work more than friendly co-workers will be more satisfied 
overall if satisfied with the nature of the work and 
dissatisfied with co-workers than if the opposite were 
the case. 

Many studies havt^- been conducted to test this suggested 
relationship between component satisfaction, importance of 
the components, and overall job satisfaction (Quinn and 
Mangione 1973; see Locke 1976 for a complete summary). 
The empirical results suugest a more complicated re- 
lationship. Evidently, individuals take importance into 
account when rat ina component satisfaction; important 
components are usually rated either very satisfying or 
very dissatisfying, while unimportant components are 
rated as more neutral. Thus, a separate rating of 
component importance is redundant with the component 
satisfaction rating. This means that a simple sum of 
component sa t i s f ac t i on ra r: Lngs is the best predictor of 
overall job satisfaction. However, note that since 
important components are rated in a more extreme 
manner, the variability in the sum of the component 
satisfaction is mostly a [product of the ratings of the 
important components. "t^lius, the importance of a component 
is valuable in format loii and does have an impact on 
overal 1 sat is f act i on , a 1 In* 1 1 indirect . 
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Regardless of the precise manner in which component 
satisfaction is combined to result in overall satisfac- 
tion, this way of viewing overall job satisfaction suggests 
that Lawler's model of component satisfaction also ex- 
plains the causes of overall satisfaction. That is, the 
employee (1) judaes for each component of the job, the 
amount that should be and (2) the amount that is received, 
(3) feels satisfied or dissatisfied as a result, and (4) 
combines all of the evaluations across components to 
determine overall satisfaction. 

However, there are two other ways to think of the 
determinants of overall job satisfaction; the person/ 
environment fit model and the met expectations model. 
The person/environment model states that a person will be 
satisfied with the job situation if it fits his/ner 
values, skills, needs, abilities, goals, and so forth 
(Dawis, Lofquist, and Weiss 1968; Seybolt 1976). Dis- 
sa' "faction results from a poor fit in either direction. 
For example, if a job is either more or less challenging 
than the individual desires, then satisfaction will be 
low. 

While the job/person fit model does not necessarily 
conflict with Lawler's model, it does produce a somewhat 
different focus. Occupational and job choice become 
important considerations, and, as explored in detail below, 
vocational education can be predicted to increase satis- 
faction if it increases the job/person fit. 

The second model of the determinants of overall job 
satisfaction focuses on met expectations (Porter and 
Steers 1975; Wanous 1977). The prospective employee has 
a set of expectations about what the job should offer. 
If expectations are not met, then dissatisfaction and, 
eventually, turnover results. This model is not as useful 
as Lawler's because the cause of expectations are not 
explored. However, the expectations model does lead us 
to consider vocational education as a form of realistic 
job preview, an idea examined below. 



P ossible Effects of Vocational 
Education on Job Satisfaction 



Using the three models of satisfaction (Lawler's 
component model, job/person fit, and met expectations), 
the effects of vocational education on job satisfaction 
can be analyzed. The discussion of possible effects is 
organized around a number of hypotheses, assertions, 
conclusions, and speculations. 
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1. Vocational education may lead to better jobs ior 
graduates, better iob outcomes, and higher 
satisfaction. 

As an example, a aood program in auto repair may allow 
a graduate to get a job as a mechanic instead of a job 
pumping qas . Comparatively, the mechanic job would probably 
offer better pay , more chal lenge , more autonomy, better 
advancement opportunities, more status, and perhaps even 
better relations with the supervisor and co-workers. 
If all of this happens, then higher job satisfaction would 
be predicted. 

This argument rests on an important assumpt ion--that 
"better jobs" are available and are more likely to be 
obtained following qood vocational education, as compared 
to poor training or none at all. If no mechanics jobs 
are available or if the chances of getting such jobs are 
not improved by vocational education, then this hypothesis 
will not be born out. An implication is that increased 
job satisfaction may depend just as much on the job and 
labor markets as in the quality of vocational education. 
Further, the likelihood of netting a good job may not 
depend much on the quality of vocational education, at 
least in the short run. Potential employers cannot 
directly judge the quality of training and so must rely 
on word of mouth, previous experience with other vocational 
programs, and experience with graduates from the program. 
In other words, the image of vocational education may be 
as important as the program itself in getting graduates 
better, more satisfyLncT jobs. 

2. Better vocational education may lead to a better 
fit between the person anci the job. 

This hypothesis mav- be viewed as a corollary of the 
first hypothesis; if vocational education leads to in- 
creased job possibilities, then presumably students can 
select the job which best fits their needs, abilities, 
rrot 1 vations , .ind so forth. However, there is an additional 
point here. Good vocational training may provide a 
rea 1 istic preview of the j ob or occupation , allowing students 
to decide if that occupation fits their needs before taking 
the job . For cxamp le , the nurs ing aide student may 
discover that job does not interest her/him as much as 
she/he thought it would after finding out, during training, 
what the job duties will be. An interesting implication 
of this f:ioint is t hat., in some cases, :* t may be a good 
outcome if an individual does not go into the occupation 
intended; if the fit between the individual and the job is 
not cioinq to be Mood , it may be best to learn this 
during traininq and not after job placement. 
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The job/person fit model also emphasizes the place of 
career counseling and job placement as an integral part 
of vocational education. Thus, a thorough evaluation 
of the effects of vocational education on job satisfaction 
would include a close look at the current and former 
students' perceptions and evaluation of counseling and 
placement. In addition, the impact of expectations should 
be assessed; for example, the perceived characteristics 
of the target job could be compared for vocational 
education and nonvocational education students. If the 
realistic job preview mechanism is occurring, vocational 
education students will more correctly perceive the character- 
istics of the job (as seen by current employers) than 
nonvocational education students. 

3. Vocational education may decrease job satisfaction 
by increasing perceived inputs and not affecting 
perceived outcomes . 

This hypothesis follows from Lawler's model of component 
satisfaction. In aeneral, we can predict that good 
vocational education will increase perceived job inputs, 
such as skills, training, education, experience, and 
perhaps job effort and performance. If perceived job 
inputs are higher, then the vocational education graduate 
will feel that she/he should be receiving more and better 
work outcomes, such as better pay, more promotion op- 
portunities, more intrinsically interesting job assignments, 
more autonomy and better working conditions. In other 
words, vocational education may increase the level of 
aspiration for most, if not all, work outcomes. The 
paradoxical conclusion is that better training may lead 
to lower satisfaction — if the heightened aspirations or 
"should be" are not matched by the perceived level of 
outcomes . 

A further implication is that this "aspiration level" 
effect should be explored when evaluating vocational educa- 
tion. One simple way to do this is to ask if graduates 
feel that vocational education should mean more and 
better work outcomes. If so, exactly what should be 
better? (e.g. , pay, promotion opportunities , autonomy) . 
Finally, did this ocuur with you; did you feel like you 
got better pay, etc. , because you were a vocational educa- 
tion graduate? An analysis of these types of perceptions 
would help explain a lack of effect or a negative effect 
of vocational education on job satisfaction. 
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4. The effects of vocational education on job 
satisfaction may be small, due to the many 
causes of satisfaction. 

Referring again to Lawler's model, there are many 
variables that affect perceived amount that should be 
and is received. Although vocational education may affect 
some job inputs and may indirectly affect some outcomes 
through job placement, there are a number of other variables 
which would also af feet these determinants of job satis- 
faction . In fact , the predictions made thus far apply 
only if all other factors are held constant; to the extent 
that this is not the case and innate abilities, age, 
effort, job characteristics, own and referent other's 
outcomes will vary and make the effects of vocational 
education on job satisfaction small and difficult to 
show empirically. 

Although there does not seem to be any empirical 
evidence on the relationship between vocational education 
and job satisfaction, Quinn and de Mandilovitch (1977) 
reviewed the literature and found no consistent relation- 
ship between number of years of education cind job satis- 
faction (except for those with a college degree versus no 
degree). Their conclusion that education provides a 
"questionable pay off" in terms of job satisfaction may 
or may not extend to vocational education. However, it 
does suggest that the hypothesis of no discernable effects 
of vocational education on job satisfaction should be 
taken seriously. 

There is another reason for predicting a smal 1 e f feet 
of vocational education on overall job satisfaction. If 
overall satisfaction is the sum of satisfaction with various 
components of the job and even if vocational education 
affects satisfaction with some aspects of the job, other 
components may be unaffected. For example, satisfaction 
with the use of valued skills may be increased, while 
satisfaction with supervision may not be affected. This 
pattern would dilute the impact on overall satisfaction. 
In addition, vocational education may increase the satis- 
faction with some components, have no effects on others, 
and decrease satisfaction with yet other components, due 
to the increased aspiration mechanism discussed above. 
To the extent that the^se effects may occur in a different 
fashion for different individuals, the overall effeci-s 
of vocational education on a sample of employees may be 
very complex and difficult to identify. 



On the positive side, the models discussed above do 
suggest important variables to control, and when possible 
to use, to tease out subtle effects. For example, if a 
large enough sample is available, actual outcomes like 
pay should be held constant. Likewise, examining the job 
satisfaction of vocational versus nonvocational education 
graduates with the same job characteristics will reduce 
error variance and make it more likely that the true effects 
will be found. 

In summary, the effects of vocational education on 
job satisfaction are likely to be complex. The vocational 
education administrator is encouraged to determine — in 
advance of any data collection--why the program being 
evaluated should increase job satisfaction, using the 
points made here as stimuli for thought. The causal 
mechanisms which are believed to be occurring (e.g., 
better job placement, increased aspiration levels, more 
realistic expectations) all imply thac ore information 
should be gathered than when measuring satisfaction 
with various components of the job. (Questionnaire items 
may have to be created, although the citations given above 
will pro'- some help.) While this type of analytic/ 
diagnost^ ■ • proach is more difficult than simply sending 
out standaj^u job satisfaction measures, the information 
to be gained is even greater . 



Additional Job Attitudes; Intrinsic 
Motivation , Job '" ^ volvement , and 
Be liefs About Work 

Job satisfaction is not the only job attitude which 
is important or for which instrurr.ents have been developed. 
Further, it may be that satisfaction is less important, 
in some ways, than other attitudes and also that it is 
less likely to be affected by vocational education. 
Several of these attitudes are briefly discussed below. 
Although instruments to measure these attitudes are not 
included in this handbook, the instruments are available 
in the published literature and the citations are given 
below. 



Intrinsic Motivation 

While this term is used in many different ways, 
the best definition is that given by Lawler (1969; 
Lawlor and Hall 1970) : 
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The degree to which a job holder is motivated 
to perform well because of some subjective 
rewards or feeling that he expects to receive 
or experience as a result of performing well. 

Thus, an intrinsically motivated individual does a good 
job not because of extrinsic rewards (e.g., pay, promotion, 
praise) , but because of feelings of accomplishment, self- 
esteem, pride in work, and so forth. 

Intrinsic motivation is clearly important for high 
productivity and quality work, and has been shown to be 
related to the health and well-being of the employee. 
Furthermore, it would seem to be an important goal of 
vocational education to produce graduates who will take 
pride in their work and feel a sense of accomplishment if 
they do a good job. 

Intrinsic motivation has been shown to be different 
from job involvement, both conceptually and empirically 
(Lawler and Hall 1970) . Instruments to measure intrinsic 
motivation (sometimes called internal motivatJon) have 
been developed and used successfully in a larr i number 
of studies (see Lawler and Hall 1970; Hackman and Oldham 
1975 , 1976) . 



Job Involvement 

Although this concept has been around for many years, 
there is not complete agreement on what it is. A very 
recent review of the concept (Rabinowitz and Hall 1977) 
suggests that the following may be the most useful way 
to define job involvement: 

Psychological identification with one's work; 
the degree to which the job situation is 
central to the person and his identity. 

Although some theorists believe job involvement to 
be a matter of individual differences (produced by early 
socialization and relatively fixed) , the more realistic 
view is that job involvement is a product of the inter- 
action between the individual and the job setting 
(Rabinowitz and Hall 1977). As such, job involvement 
may also be seen as a desirable and reasonable product of 
vocational education because of the "professionalism" 
implied by the definition that "the more one is involved, 
psychcloqically , in one's work, the higher the standards, 
the greater the concern, the more internalized the values 
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of the occupction, and so forth." Vocational education 
might encourage job involvement by showing that an oc- 
cupation is important, has an impact on the lives of others 
and society in general, demands skills, and, in general, 
is worth identifying with. 

Although there is no one standard measure of job 
involvement, several validated instruments exist. These 
are described and critiqued by Rabinowitz and Hall 
(1977). Citations for the instruments themselves are 
available in that article. 



Beliefs About Work in General 

An important result of vocational education may be 
a change in underlying beliefs about work in general 
and the part it should play in our ""ives. Very recently, 
Bucholz (1978) has reviewed the literature on beliefs 
about work and identified five different belief systems: 

1. The work ethic. Work is good and bestows dignity; 
na_rd work leads to access. 

2. The orga .izatic:: : f-c^1.ief system. Work takes 

on meaning only js i'c affects the organization and 
one's statu-' within the organization; success 
comes from conforming to group norms . 

3. The humanistic belief system. Work is how people 
fulfill themselves; jobs should be redesigned to 
satisfy higher-order needs. 

4. Marxist-related beliefs. Work is necessary for 
fulfillment, but the capitalistic system does not 
al low this , because workers have no control . 

5 . The leisure ethic. Work is necessary , but can 
never be fulfilling; because fulfillment comes 
from leisure activities, work should require 
less time than it does. 

Which belief system should result from vocational 
education is a very value-laden judgment. However, it is 
likely that most administrators would identify the work 
ethic and perhaps the humanistic belief system as goals. 
In any event , Bucholz has created and validated an 
instrument measuring each belief system, so it would be 
possible to evaluate the effects of a program on these 
core beliefs about work. 
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Job Perforr^nce 



The Dimensions of Job Performance 



If anything, job performance is even more complex 
than job satisfaction; it is not really one concept, but 
a whole cluster of dimensions. When job performance is 
measured, it is usually assessed along some number of 
dimensions, which often include such things as the 
following: quality, quantity, skills, technical knowledge, 
ability to work without supervision, attendance, punctuality, 
cooperation , communication skills , problem solving skills , 
teamwork, interpersonal skills, safety, good attitudes, 
and initiative. The list could go on forever. 

There is no one correct list of job performance 
dimensions for all jobs. The ideal approach is to do a 
complete job analysis for a specific job and derive an 
instrument for that job. At the other extreme, a small 
set of very genera 1 dimensions could be applied to any 
and all jobs. A manageable middle ground is to select an 
instrument specifically developed for the job being 
assessed or a similar job. Another approach is to select 
an instrument which contains dimensions believed to be 
meaningful, as judged by someone familiar with the job. 
Using the large number of instruments collected in this 
handbook, the latter approach seems quite feasible. 

In the discussion that follows, job performance is 
discussed as a qeneral concept, witnout reference to the 
many dimensions of performance. However, it will become 
clear that many' of the commonly used dimensions of 
"performance" are actually measures of either motivation, 
skills, or role concept, which are factors that lead to 
a qiven level of performance. 



A Model of Job Performance 

Job per fori.. aiico is Generally thought of as determined 
by three basic variables: motivation or effort, skill 
level, and role conception. Elf fort (how hard the employee 
works) and skill level (whether or not the employee has 
the skills to do the job) are sel f -explanatory . Role 
conception refers to the employee's idea about what should 
be done on the job--how the role should be played. If 
an employee has an incorrect notion of how the job should 
be done or what acti/ities constitute high performance, 
then actual performance will be low; even though the 
employee may have the abilities and be working hard, 
[performance suffers if effort is put into the wrong 
activities . 



These three determinants of job performance are thought 
to combine multiplicatively in affecting performance. 
That is, if one variable is very low, then performance 
will be low. Total lack of job skills means low per- 
formance, no matter how much effort is put into the work. 
If the motivation is lacking, then performance will be 
low, no matter what the skill level or role concept. In 
order to analyze the possible effects of vocational educa- 
tion on job performance, we must determine which of these 
three factors might be altered by vocational training. 



Motivation and Vocational Education 

There are too many theories of work motivation to 
summarize all of them here. However, there are two 
major approaches to motivation that dominate current 
thinking and provide a basic framework for analysis. 

Expectancy Theory . The dominant theory of work 
motivation currently is expectancy theory (see Campbell 
and Pritchard 1976, for a comprehensive review). In 
simple terms, expectancy theory states that an employee 
is motivated to perform if effort is seen as leading to 
performance, if performance is seen as leading to work 
outcomes, and if those outcomes are desired by the in- 
dividual. Work outcomes may include extrinsic outcomes 
(e.g., pay, promotion, relations with co-workers, worKing 
conditions) or intrinsic outcomes (e.c,., autonomy, 
feeling of accomplishment, pride in work, enjoyment of 
the work itself) . 

Thus, vocational education could increase motivation 
by affecting any or all of these three expectancy theory 
components. First, the perception that effort leads to 
performance might be affected; as students learn how to 
do the work and become more confident, then they will 
begin to see that high performance can be attained if the 
effort is given by them. The perceived relationship 
between performance and work outcomes seems less amenable 
to shaping by vocational education. This perception is 
affected primarily by the reward system in the organiza- 
tion itself; some organizations reward performance and 
others do not. The final variable, the value placed on 
work outcomes, might be affected by vocational education, 
for some outcomes. The value of intrinsic outcomes 
(e g , autonomy, pride in work, feeling of accomplishment) 
seems more likely to be affected than the value of ex- 
trinsic outcomes (e.g., pay, praise). In summary, 
expectancy theory might predict a slight effect of 
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vocational education on intrinsic motivation to perform^ 
if it increased the perception that effort 1pads to per- 
formance and the value of intrinsic wor}. outcomes. 

Goal Setting > A si. iqhtly different approach to 
motivation has been advocated by Locke (1968) . He suggests 
that performance is determined by the goals a person sets 
for hi mself /hersel f . Further, the more specific and the 
more challenging the qoal, the higher the level of per- 
formance. Finally, the individual must fully accept 
the goal before it will affect performance. 

Using this approach, vocational education may increase 
motivation and performance by altering the performance 
goals of the individual. More specific goals may be 
acquired during trainincj, as the student is taught exactly 
what doing a good job entails, More challenging goals 
may emerge as job skills increase and aspiration Ic -els 
increase. Goals will be accepted more completely as the 
student understands the reasons behind performance 
goals and identifies with the occupation. 

In summary, vocational education might increase 
motivation and consequently job performance by (1) increasing 
the perceived probability that effort leads to performance, 

(2) increasincj the value placed on intrinsic rewards, and 

(3) increasing the [performance goals set by the individual. 
Unfortunately, many o^her variables also affect motivation, 
such as job characteristics, supervisory style, organiza- 
tional reward systems, value t)laced on extrinsic rewards, 
and work nroup f)er f ormanc*' norms. Since these are largely 
unaffected by v'ocational (-ducation, the overall effect of 
even high Ljuality vocational education on work motivation 
may be a moderate one. 



Skills and Vo ca t ionaj I-:duc at ion 

The second variable affecting job performance is the 
skill level of the individual. Vocational education is 
generally thouglu. to ruive an impact on job performance 
primarily by affect in^.; this variable. Although this may 
be true, the situation becomes more complex when analyzed 
closely . 

Fawler ( 197 3) iias sugues ted that job skills result 
:rom underlying aptiturles, training, and direct job 
♦.'Xper i ence . Because aj^ticudes are generally considered 
to be essentially stal;]e characteristics, vocational 
education would not be expected to affect job skills 
^hrouqh aptitudcvB. Likev-'if^e, experience on the job 
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affects skills apart from the quality of vocational 
education. Even within the remaining "training" category, 
vocational education is only part of the story. Important 
skills are certainly affected by the quality of non- 
vocational education , hobbies , extracurricular activi ties , 
and other nonschool experience. All of this is to say 
that even though vocational education may affect job 
skills, it is only one of many factors that affect such 
skills. Therefore, the effects of vocational education 
on job performance through job skills may be modest and 
difficult to find. 



Role Conception and Vocational Education 

This is the least studied and least discussed of the 
three causes of job performance. Yet it is significantly 
different than the other factors and probably has a large 
impact on job performance in many situations. It is not 
a matter of being motivated to perform well or of having 
the necessary sk ills ; rather , the idea about what the 
job entails differs from the correct model or the one 
advocated by the organization. For example, the job 
performance of an insurance agent may be rated low if he 
feels that her/his job is to service existing clients and 
not to develop new customers. 

Vocational education may have an impact on this cause 
of job performance; part of good training may be learning 
what the occupation entai ls--which facets are most 
important and which of lesser priority. If this effect 
of vocational education occurs, it is likely to be most 
noticeable for new employees. Assuming that most un- 
trained employees only learn the proper role concept over 
time as they gain direct job experience, the graduate of 
vocational education i^tiould be more able to step into the 
job and know what tasks should be done with minimal 
direction . 



Overall Conclusions 

1. There are many causes of job performance . 

This seemingly obvious point must not be forgotten , 
for it implies that the effects of even very good voca- 
tional education will be diffxcult to discern because of 
"error variance." If all the extraneous factors could be 
controlled (e.g., individual motives, job characteristics, 
organizational reward systems, underlying aptitudes. 
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job experience, other training), then the effects of 
vocational education could bo easily and unambiguously 
determined. In any real life study, these uncontrolled 
factors mav create so much "noise" that no effects can 
be seen. 

Several suggestions can be offered. The use of large 
numbers of subjects can help to tease out subtle effects. 
Whenever possible, the factor suggested by the model 
presented here should be measured and controlled for, 
For example, comparing vocational education graduates 
against nonvocationa 1 education graduates within the same 
job classification, organization, and length of service 
would help control many factors. 

2. There are several ways in which vocational educa- 
tion may af feet job f:>er f ormance • 

On a brighter note, this analysis has suggested 
several possible mechanisms through which vocational 
education could af feet job performance : increased 
motivation, better job skills, and knowing what the role 
entails. Whether or not these things do occur is, of 
course, an empirical question. Flowever, recognizing the 
possible causal mechanisms has two important implications. 

The program itself should be analyzed for existing 
effects and possible improvements in the motivational, 
ability, ana role concept domains. The first step is to 
examine the goals and conduct of the program, Specific 
lob skills are certain^^ a central focus, but is it also 
the intent to increase motivation and knowledge of the 
role? If so, the discussion above provides sor^e specific 
Guidelines for analysis (e.g., is self-con fideace or 
f)erceived effort to performance relations enhanced, are 
high fjerf ormance standards encouraged, is the in.portance 
of the occupation emphasized, is autonomy encour„:ged?) . 

A comprehensive, useful evaluation of job performance 
of vocatLonal education graduates will not be limited to 
measurincf overall performance; items should deal with 
motivation, ability, and role concept. Whatever the 
findings for over a 11 job performance , the administrator 
will need to know why there was or was not an effect on 
f rformance. Further , the implications for the content 

conduct of the program can only be known if more 
specific items .ro included. Imagine that a program 
is evaluated by rnoasurino the job performance of recent 
cjraduatos and that they are found to be performing no 
better than similar cmploveos without vocational training. 
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What should be done? A likely response would be to try 
to add to or update the content of the program — that is, 
try to increase job skills. However, the problem could 
be low motivation of graduates (e.g., the teacher lowered 
student's self-esteem and perception that effort will 
lead to performance by constantly criticizing) or role 
concept (e.g., the teacher has an idea of how the job 
should be done that differs completely from that of the 
eventual employer). In other words, evaluation of 
only the overall job performance of graduates may actually 
be worse than no evaluation at all; without more specific 
information on motivation, skills, and rolo concept, the 
wrong problem could be solved. 

In conclusion , the cluster of concepts known as job 
performance are complex. However, by (1) measuring 
dimensions of performance which are important for a given 
job and (2) measuring the causes of performance — motivation, 
skills, and role concept--a useful and understandable 
evaluation can be performed. 



Th e Research Design: How to Assess 
the Effects of Vocational Education 



Along with selecting instruments to assess jol satis- 
faction and performance, the vocational education ad- 
ministrator must decide upon a research design to evaluate 
the effects of the progrv^m. While there are too many 
issues to deal with here, a few suggestions can be 
offered . 



The Ideal 

The perfect design would involve conducting a true 
experiment. Students woald be randomly assigned to voca- 
tional versus nonvocational education and then put in 
identical jobs in the same organization. Any differences 
between the ^wo groups on satisfaction or performance would 
clearly be caused by the different educational programs. 
Although this research is impossible to carry out, it 
does highlight the possible extraneous differences between 
a vocational education sample and any comparison group. 
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The Alternatives 



Compare wi th co-workers . The job satisfaction and 
performance of a qroup of vocational education graduates 
could be compared to that of their co-workers in similar 
jobs in the same organization. This would entail selecting 
(or having the employer select) one non vocational educa- 
tion graduate for each vocational education graduate in 
the sample. This approach would begin to control for the 
effects of job and organizatio.ial characteristics, but 
individual characteristics (values , innate abilities , 
age, experience, etc.) would not be controlled for, due 
to nonrandom assignment to groups. Even with these 
problems, it is the best of the feasible research designs. 

C ompare with nonvocat ional education co ntemporaries . 
Another comparison qroup could be school-mates who did 
not select vocation a 1 education . This might control for 
some individual characteristics (e.g., age, years on the 
job, socioeconomic status, certain values), but would not 
deal with the very impor tant factors of job and organiza- 
tional characteristics . In addition , there may be im- 
portant di f f erences between the two groups which caused 
their choice of educational programs. The advantage of 
this design is that it might be easy to carry out, and 
it may provide some useful information if combined with 
the first design described. 

Compare wi th " norms " published with the instrument . 
A few of the standardized measures in this handbook provide 
data from large groups of employees in different job 
classif icit ions or demographic categories (e.g.. Job 
Descriptive Index, Minnesota Satisfaction Questionnaire) . 
Although these norms are often several years old and may 
not match the characteristics of the sample being studied, 
such a comp-arison is inexpensive and provides some useful 
in format ion . 

^ ^^ggost ion . When a perfect research design 

is not available, the best approach is to combine as many 
limited designs as jjossiblc. Therefore, if possible, all 
of the designs discussed above should be used. Because 
each has different strengths and weaknesses, a combined 
strategy will lead to results that can be accepted with 
some con f idence . 
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REACTION TO JOB SATISFACTION 
AND PERFORMANCE MEASURES: 
THE STATE OF THE ART* 

I must admit to th is audience that my initial reaction 
to this paper was one of frustration. It bothered me that 
the writer did not expand on the excellent framework set 
down. I felt that he should have taken time to run it 
by someone representing a large urban area. By doing this, 
the writer could have qained some rather specific examples 
to back up the points raised in the paper. 

After several more roviews and note taking on my 
part, I realized several things. First, that the National 
Center for Research m Vocational Education established 
a series of cruides that [jretty well set the pattern for 
the presenters. Second, to deal specifically with uhe 
large urban areas would, in turn, leave out about 40 
percent of the vocational education population. Third, 
that the subject matter is as large as the number of 
titles found in the Dictionary of Occupational Titles 
and the people that fill those positions. Fourth, although 
it is my understanding that the various state departments 
of education will establish the procedures for determining 
job satisfaction and performance, it would be most helpful 
for vocational staff to bo able to have a more viable 
comprehension of the subject matter. 

In reviewing the pa:^or "Job Satisfaction and Per- 
formance Measures: The State of the Art" by Rcb^^rt S. 
Billings, one must consider the possibility that the 
vocational educatior: system is, in part, a servant of the 
business-industrial-labor complex and, in part, a servanv 
of the individuals enrolled in the programs. Regardless 
the intent of Congress, tlic will of the state or local 
boards of education, or the dedication of the delivery 
system staff, the final results of the evaluation process 
rest upon the new entrant to the work force and his/her 
employer. 



* Redaction by Ijonalii H.-alas to Robert Billings' 
present..^t ion . 



The dynamics iiivolved in being able to provide a 
reasonable response mechanism between two strong in- 
dependent entities, the student/graduate and the employer, 
is awesome. In one field, you have a graduate/student 
who: may or may not elect to go to work; in an occupation 
that may or may not be related to his/her vocational 
program; may opt for a single-skilled occupation versus 
a multi-skilled oc-upation; or select a lower paid 
position versus a higher paid position. 

In the other field, one finds employers who: may 
find it easier to import skilled individuals versus 
employing and preparing a new group of individuals; may 
not be hiring because of the economics of the area; may 
be concentrating on meeting federal mandates for new 
hires; may be involved in an upgrading program for present 
employees; or because of negotiated contracts, have to 
select workers for skilled occupations from a general 
labor force within the company. 

The paper attempts to provide a set of relat-^^d 
statements and questions impacting on a critical segment 
of interpreting outcome measures for vocational education. 
In addition to responding to an extremely demanding subject, 
the writer had to be aware of his target audience. The 
makeup of this audience has probably caused more than its 
share of concern for all of us who know it and are involved 
in providing responses for the present and future reporting 
systems . 

The chapter has been developed with a great deal of 
insight and brings to the attention of the reader a 
series of management concepts relating to job satisfaction 
and performance. It is general enough to cover the salient 
points for the possible users with some experience in the 
field of vocational education. However, to those without 
the experience and/or awareness o£ the dynamics at work, 
the paper may present a rather well-documented series of 
possible prescriptive statements without reader considera- 
tion for the variables highlighted by the writer. The 
paper, divided into four sections, places ma j or emphas is 
on job satisfaction, somewhat less emphasis on job 
performance, and only a cursory look at additional job 
attitudes and the research design. Although the writer 
touched very briefly on student/graduate self-concept 
by implication, it would seem to warrant either a secti 
by itself or to be made more visible in the two major 
sections . 
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In the first section, job satisfaction, the writer 
zeros in on Lawler's work, summarizing and evaluating 
the various models of the causes of job satisfaction. 
He has also cited other national works and findings 
which support the projections and questions raised for 
vocational educators on the possible effects on job 
satisfaction . 

In addition to Lawler's component, satisfaction, the 
reader is introduced to the job/person fit and the met 
expectations , mode Is of job satisfaction . The writer 
then proceeds to investigate the potentials of var ic us 
components using four questions relating to vocational 
education . 

The job performance section covers the general 
dimensions of performance measurements while pointing 
out the depth of difficulties to be found in an ideal 
system versus a [^racfmatic approach. The variables of 
performance may be classified under either motivation, 
skills, and/or role concept according to the information 
contained in the chapter. The expectancy theory and goal 
setting are reviewed under the motivation heading. 
Role concept has also been explained in some detail. 
The writer points out that it is the analysis of data 
received that provides for the improvement of the de- 
livery system. 

The two remaininq sections, additional job attitudes 
and the research design, have not really been fleshed 
out. The firsi of these covers a brief summary of 
intrinsic motivation, job involvement, and beliefs about 
work in general, all of which are helpful and could have 
added to the whole. The second part provides an overview 
of possible designs from the ideal to alternatives and 
proposes the best combination of the alternatives. 

The evaluation of job satisfaction and performance 
is, at best, a complex maze. The variables are as 
numerous, at least, as tne number of individuals involved. 
While job satisfaction and the related attitudes may be 
reviewed through instruments utilized by the students/ 
cjraduates, it would appear to be a m.ost difficult task 
to complete a job performance evaluation and match it to 
the satis faction responses , if indeed that is required . 
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SUMMARY 



1. The material contained in this chapter is important 
to those involved in the collection; interpretation, 
and dissemination of job satisfaction and performance 
data . 

2. The achion-orientcd users will need to brush up on 
management concepts and add the local inputs , 
variables, and components as required. 

3. What provisions can be made for tl».^ s tudent/graduate 
who selects an occupation that requires a general 
labor force r.ntry and then seniority for bidding up 
to skilled work? 

4. What provii )r.3 can be made for the students/graduates 
who complc':^^'- a vocational program and 

a . changed ttujir occupational objectives ? 

b. elected not to go to work? 

c. c'lccteJ to take a "low' paying job? 

i. elected to take a single-skilled job rather 
than a mul t i -ski 1 led job? 

5. How can we determine the student's priority factor 
in selecting his/her job? 

G. Are studen tb/graduatos entering the work force for the 
first time able to make the required responi:os in a 
logical manner to a rather sophisticated questionnaire? 

7. Have the studies cited in the paper included the student/ 
graduate entering tl'ie work force for the first time? 

8. How will the Occupational Work Experience and/or the 
Diversified Cooperative Health program student be 
counted in the related training category? Both are 
found in clusters of occupations, not single occupations. 

9. What percentag :)f all returns will be acceptable? 

It would seem that a universal system of evaluation for 
a not-so-un i versa 1 vocationa 1 del i very system is a 
mathematical jigsav of staggering proportions. 



CRITI-R IA AGAINS T WHICH VC C/v TIONAL 
EDUCATION SHOULD BE' HELD ACCOUNTABLE* 



The long recoanized success of American education 
has caused most educators and citizens to rely on 
accountability criteria with .vhich they are familiar, even 
though the future which the learner must face today is far 
different than it was in the past. It is also true that the 
skills, knowledge and experience needed in the work world 
of tomorrow require more and different competencies chan 
were adequate in the past . 

Recent studios by the National Academy of Sciences 
titled Assess i ng Vocat ional Education Research and 
Development ^ ; by Project Ba^s eline at Northern Ari zon a 
University titled R e port to the Nation on Vocational 
Educat ion 19 7 5 ^; by the Panel on Youth of the President's 
Science Advisory Committee cal led Youth ; Transition 
to Adul thood 3 ; by the National Manpower Institute of 
Washington , D. C. ; in a book called The Boundless 
Resource : A Prospectus for an Education/Work Policy ^ 
and others related to the problems, issues and priorities 
of preparing individuals for a future work life all 
indicate the need for now criteria against whicL to 
evaluate education of all kinds and especially vocational 
educat ion . 

Traditionally, criteria for accountability in regara 
to th<:: [^reparation of youth for the future, including the 
work role, have been determined over a long period of 
time based on experience, wisdom and judgment of the 
society power structure. This method has worked well, 
primarily because the future was like the past and an 
understanding of the past w^as the best way to determine 
what the future would recjuire. The problem of "what to 
measure" vjor^ relatively simple as compared to todav* 

It was oven more likely that the criteria of ac- 
countability for vocational education was relatively 
simple wi.th work and jobs remaining the same over 
generations and for most people being priniarily manual. 
The same could r:>e said for nearly all of education in 



Pr'isen tat ion by c^.rar^t Jdnvt, 



terms of ■ .static. Ihms. '-.ho moasuremont of succe i 

in meeting L.;c-od rather .'..imtJlt' and static criteria became 
the isolated effort wl a i ^.v ;. rof essionals and a few 
dollars and was not part of the main concern of education, 
nor considered highly significant by most educators, 
citizens, or policy makers. The amount of money spent 
and past efforts indicate the deqrce to which the selection 
of criteria p.gainst which vocational education was to be 
held accountable was not a significant evaluation priority. 

Thus, a concern for methodology for measurement became 
more significant thari the se lection of criteria against 
which to measure. Without even implying any criticism, 
it is fair to say that those in the field of research and 
evaluation found themselves forced to deal with short term, 
small parts of the education process rather than broad 
policy, direction or criteria setting, or even the broad 
question of accountability criteria. 

The long and continued isolation of education from the 
rest of society's daily pressures and conflicts, especially 
the education-work relationship, forced the practitioner 
to select criteria for success in vocational education 
(placeiiient in a training related job) that was not of a 
great concern to the rest of education and specifically 
not to specialists in the field of educational research 
and evaluation. 

Very little vocational education research, development 
or evaluation exists outside the overal.l educational 
research and development effort in the country. The 
history of research and development efforts at the 
federal level in the United States Office of Education 
illustrates the point. Today, direct work and 'Ob related 
efforts are probably more clearly identified with the 
Department of Labor than with the Office of Education. 
Even the most recent reorganization of the National Institute 
for Education (NIE) points again to the fact that for 
whatever reason the relatioi sliip between educational 
research, evaluation and measurement and success criteria 
in vocational education has been tenuous at best and is 
still insignificant in research area priorities, which 
is suggested by the NIE reorganization. 

I is also true that the dominance of physical 
science research methodology, where variables can be 
controlled and outcomes today are as they were 1000 years 
ago , often causes educational research to keep searching 
for the "right way." There is a "mystique" that "good," 
"st)lid," and "respectable" efforts presently in use will 
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Gvontually triu^^n.h ar;ri tluit the "r-iqht criteria" against 
which vocational education should be held accountable are 
similar to past criteria. The p. jblem is seen as a need 
to simply (lather rho ri^ht ^lata, analyze it properly, and 
draw conclnr»ions which v; i 1 1 obviously result from such 
evaluation. IL is also Iikoly that many pract ictioners 
feel that all the fuss about accountability and the new 
legislation compliance reeiui rements are figments of 
somebody's imaaination anci in. no way related to the real 
world. This does not im[)ly tli.it whatever evaluation is 
done can bo dono v/ith.out sound [processes. 

Mowevcr, it. may i)^' that the search for evaluative 

I't'itoria that are n(.)iichaiuivjablo may become the search 

that IS unenduM and could load to the cul-de-sac that 

methodology has or ton on L*^* rod: if we can't measure it, 
It isn*t s icfni f icant 1 

While thorr- um;: \)0 no ^.-ortamty that any criterion 
chosijn can standi t Ml* t^'St of tim.e , it seems certain 
that tjresor:t crLr^Tia ^iro inadequate, often conflicting 
and not clearly d(M'i:v^d at any level, federal, state or 
local. The criteria that are often used by evaluators 
are sometimes r-art o: the resistance to crec. ing new 
organization, policies, and objectives that could help 
make vocational '^'ducation more viable, flexible, and 
relevant for the learner, the taxpayer and the employer. 
Thi.) r'oason for this statement is the fact that as the 
criteria for accountability change it will be difficult, 
if not imposs .i b 1 f; , to find the most valid and reliable 
mc^thods of moasurinu succ^f -ss-'thus , it becomes important 
to not chana*-- fh^* criteria t rom the standpoint of the 
ovaluator. The rec- rit and continuing national reaction 
to failing SAT scores indicates the problem. Even 
though the criteria wore chosen to predict success in 
(^ollogo, and wore- predicting successfully, many persons 
arcjued that the \\\qh schools must go back to their one 
[)urpose--[;rer;a r i ng youth to enter college. Multiple 
critc^ria and chaiujina criteria m.ake the job of evaluating 
success difficult to do and difficult to explain. 

Even more troubl osoni*.? are multiple processes by 
which vocational education criteria are established. 
These j^>rocessos are »^ven m/3io complex than those in 
th<-) rf:'St of education, for the following reasons: 

1, S.fjcific <.atr:omo n'oasures and operational criteria 
ar S(^t 1)/ fccioi'al law and regulation. This may be 
t he reason why • "ocat ional administrators feel their 
most im[H)rtant c n J •: ime consuming tasks are their 
efforts to comnl\ with federal and state regulations 



2. 



Empl-oyeGS that hire directly from the vocationally 
educated student bcdy have specific and dofinitive 
criteria defined by occupation and by job. 



3. The economic and social neecis of the nation may 
be more directly or immediately affected by the 
success or failure of vocational education 
both in terms of meetinq manpower demands and 
reducing unemployment and emergency expenditures 
than by most other education programs in the 
schoo Is . 

4. The different methods and equipment used in 
vocational education are often irritants, if not 
direct threats to the rest of the educational 
establishment , at all levels , and thus criteria 
for success in vocational education often ge t 
changed to reduce these conflicts rather than 
defined more directly to measure vocational 
success . 

5. Those directly responsible for vocational education 
;>olicy and objectives are often not familiar with 
either vocational education or the needs or problems 
of the work world and thus tend to support their 
own concepts of quality. 

6. The historical separation and development of 

work and education has created a chasm which, even 
today, is seldom bridged when most people think 
of either education or work. 

Tnese few examples are given to point up the inherent 
and difficult problem of establishing criteria against 
which vocational education 5=hould be held accountable that 
will be acceptable to the educator, the employer, the 
legislator, the vocational .-^Jcatcr and, perhaps most 
important of all, to the r >irearcher and evaluator. 

Allow me, for purposes of this paper, to redesign 
Glaser's "Ten Untenable Assumptions of College Instruc- 
tion, "6 to fit the teacher in vocational education whose 
criteria are set by everyone, constantly changing, and 
now measured against a sinale iactor over which he/she 
has no control, the job market. 

Of cen the vocational teacher is forced to act as if 
the arsumptions listed below were t^ue»* 



1. The specific job knowlodqe to be learned 

by the student will relate to the student's 
future work and career plans even though the 
career r^lans of the student are^n't known. 

2. All students in each course come with equal 
aptitudes, loarninqs and experience. 

3. Aptitude is more important than previous achieve- 
ment and motivation for the attainment of skills 
and knowledge . 

4. All people learn in the same way and take the 
Game time to learn th^^ same amount of the same 
thing . 

Listenincj to lectures, reading materials and 
learn inc; rules are [.powerful means for changing 
learner behavior . 

5. StU'ients retain knowledge and skills without 
much rev i ew or use , 

7. Grades tell what a student has learned and can do. 

8. The vocational teacher is, by virtue of a teacher' 
authority, able to teach the right things well 
and knows what should be taught . 

9. Vocational education instruction is enough to 
teach a student how to make the transition to 
work and r*:'F.ponsibi iity , 

10. The organization and structure of vocational 

educatic^n as it complies with federal and state 
regulations is the best system for transmitting 
knowlerlqo anci skills to students at the local 
leve 1 . 

F::ach of us woula quickly realize that not all voca- 
tional education instructors believe the assumptions, yet 
the environment in whi^'h most vocational education is 
carried on and the educational system structure, organiza- 
tion, money and j riorities all tend to force compliance 
with traditional educational success criteria and evalua- 
tion methods, even though a look at the future relation- 
shifts between education and work would suggest some new 
criteria acjainst which vocational education should be 
held accountable . 



One thing scens ce rtain--sctt incj criteria for 
accountability and then measurLng outcomos arc not 
consolidated efforts in vocacional education. At the 
present time, the federal government, through legislation, 
regulations, and categorical funding; the- state through 
administrative program plans and fund allocation; and the 
local units by interest, compliance or nonconcern, create 
the variety of criteria which the individual school and 
teacher mus^ neet and upon which they are j udged. Yet 
these all f ^v. ^ into the background as the individual 
student, parent, teacher or employer comes face to face 
with the question, "Is this person able to be successful 
today and in the future in his/her private, public, and 
work 1 i fe This concept of account abi 1 ity may be a 
more significant basis for attempting to determine a new 
approach to setting criteria because it is truly the basis 
upon which each person judges the worth of a social 
organization, an educational institution, appropriation, 
or specific program. What is so significant is that 
change has become so pervasive, especially in the area of 
work and employment, that it is fair to say that everyone 
must be educated i ^ order to work or society will be forced 
to care for them. What may be even worse in our culture 
is that those who do not work have neither worth nor 
dignity . 

Thus , criteria against which vocational education 
..'ist be be"^d accountable in the short and the long run 
must come Lrom the individuals in the society into which 
the student must pass and to whom the student must demon- 
strate that he/she lias learned and can learn and that whnt 
he/she has learned he/she can apply as judged by others 
tnan those in vocational education , education , or e .alua- 
t ion . 

Both process and product revaluation will be made 
regarding vocational education by the public, since eacl 
person nas had experience in education. If this experier; . 
has been successful, as it has for most "power structure 
people , " both with in and outside of education, the degree 
to which criteria will be "right," as seen by these people, 
will relate to their successful experience in education. 
However, today, youth unemployment and isolation; lack of 
career goals ; f ai 1 ing SAT scores ; and growing wel fare , 
education and crime costs all con^:ribute to people's 
fears regarding the quality of e(/u':ation and the quality 
of vocational education. 
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In searching for* new accountability criteria that 
relate to individual development, sc cietal needs and 
future change, one must cons.-.der some of the questions 
that need to be asked in settinq priorities for new 
efforts in evaluatinq 



1. What arc some ^liffcrent things 'hat need to be 
done as contrasted ^:o the constant question of 
how to do bet ter what we are already doing? 

2. Fiow can wo bo note concerned with change rather 
than riqht answers when we know that much knowledge 
and skill has a short life in a technological, 
changi nq work force ? 

3. How dc we come to value creativity in the 
d^:*velopmen t of new methods, organization, and 
f)u»".^oses over compliance in vocational education? 

4. Fiow do w*.' find criteri.^, against v -.ch to evaluate 
vocatioHc:! education that are more useful than the 
traditj-onal criteria of preparation for placement 

in a job related to training? 

5. Can vocational education demonstrate new ways to 
learn to all of education rather than mimic other 
educat ion ? 



6. Do vocational educators owe first allegiance to 
the consumer rather than the profession, in the 
area of accountability? 



There are many other questions that could be raised 
but the sense of the r)roposition ±s that changes in our 
society have become so qreat, especially as it relates 
to education and work, that new approaches are more 
important than merely improving or creating minor changes 
in a system based on the ;)ast rather than the future. 

Perhaps <:\ brief listirui of a few of these changes that 

support the [^remise thiat a major new look at criteria for 

accountability is more important than simple improvement 

of criteria measurement :>r minor additions to the li.st 
of ::riteria would r^rove helpful. 



F undamental Societal Change s that 
Demand New Accountabi ^ i cy Cr i torTa 

Changes in the Nature of Work 

Much of what was needed to be a mat ure working adult 
regardless of valents^, interest or work opportunities 
was lea meet through experience in the home , ne ighborhood 
or work pl^ce; most youth no longer have this opportunity. 

Changes in Criteria for Su cce ssful Adulthood 

The fading work ethic, mobility in the work place, 
changes in family structure and role all imply new ways 
to prepare for adulthood as part of education- 
Chan ge^_ in t he Naturo of Educ at ion and Schooling 

One may need to become more a learner through the 
iucation process than to become a "learned" person, 
:c complete an education is more myth than reality. 

More Complex Passages and Transitions in 
Private , Public and Work Life 

To prepare for a vocation with the expectation that 
the knowledge and skills will be good for life is no 
more realistic than to expect that one ^3 likely to live 
in the same place or in the same manner all one * s life 
The transitions in adult life will require different 
educatioi'. than did a stable unch nging past, especially 
in one's work life. 



The Futur e as Different f rom Toda y 

The most challenging societal change for which 
vocational e^. cation must help prepare the individual is 
the fact that we cannot tell exactly what the future v/ill 
require. What accountability criteria are necessary that 
will predict success when no one can pT-eo.ict what will 
be needed? This has occurred more of cf n ;i the work area 
than any other. 
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Broad Caieqorics or Evaluation Criteria 



In attempt inq to suqqest new criteria for evaluating 
vocational education, on^.^ mu:). recognize that there are 
new fundamental, social, and ...dividual needs that can 
best be met by ostablishinn new objectives or criteria 
against which vocational education should be held ac- 
countable. The question of how to measure or the methods 
of evaluation should not be considered in the original 
search for better criteria. 

If t^ie ciuestion of "hew to evaluate" gets into the 
process of settinq critL-ria, as do the comm.on arguments of 
"how to meet the obi-ctives," too often critical needs 
are overlooked because present instruction and organiza- 
tion j^atterns and evaluatioix techniques were not designed 
to be responsive to the new objectives. Therefore, this 
paper will not consider the question of instructional 
process or evaluation techniques, nor will it in any way 
attempt to demonstrate the validity of the new criteria 
by giving examples of the use of the criteria or examples 
of how and where they have been measured. This, it seems 
to me, is the job of the researcher, the evaluator, and 
the instructional expert whose expertise is in the area 
of instruction and measurement. The setting of criteria 
for purposes of instruction and evaluation is primarily 
a value judgment process which must stand the test of 
publ ic acceptance . 

The previous parts of the paper have attempted to 
describe the reasons for needing new criteria in addition 
to indicati nq some or the specific present prnctices that 
are setting objectives in vocational education, and the 
limitations of som^- of the evaluative processes that tend 
to i)revent selectivjn of new criteria. 

The selection of the four broad categories of criteria 
is based on premises that assume the primary role of 
education in fjreparing for the future. 

1. Education has is its major purpose providing 
spec i f ic know led qe and skills as well as learning 
opportunities that will help the individual prepare 
for the future. This is also the purpose of 
vocational education . 

2. Education is a societal necessity in preparing 
for the future ar^d preparation for work has 
become cessar t ~ r everyone in a technological 
soc iety . 



I b 



3. Education must be responsive to the unique 
differences aiaong individuals and among groups 
in order to educate them for future societal 
roles, including work. 

4. Education must be rosponsi.ve to change as a 
fundamental component of the future , especially 

in the area of work, and thus vocational education 
must design objectives which brings change in 
program structure and purpose as well as in the 
environment where vocational education takes 
place . 

5. Even if education meets individual needs, 
responds to societal needs , and meets these criteria 
efficiently , there is still the question of the 
value of these outcomes if the individual has no 
opportunity to use his/her education in a work 

role and society, therefore, receives no benefits 
from its investment in vocational education. 

6. Vocational education has a specific purpose 

of meeting societal needs in the area of developing 
v;ork skills especially in the areas where new 
work skills are being created. 

7. All learning for the future cannot be done in 
the school and the emerging skills, knowledge, 
and understandings learned through vocational 
education are often originally developed and used 
outside the school, thus the involvement of the 
community becomes fundamental both to learning 
and also to establishing new criteria. 

The broad categories of criteria suggested in tne 
paper derive from the implications of societal change and 
the recognition that the most common program objectives 
and processes were established in vocational education 
long before change became the most valid descriptor of the 
future. They also assume that regardless of how well 
vocational education can measure and prove that it is 
meeting its objectives that many persons question the 
objectives or criteria in use. The categories chosen also 
reflect recognition of the fact that change has created 
much confusion as to just what vocational education 
should be. 

The categories selected provide the common dichotomy 
of evaluative criteria ij.to process and product categories 
since the two are necessary components of the teaching and 
learning process. 




Quality of Instruction , Th is category was se* ^cted 
because It Is fundamental .aqardless of objectives and 
regardless of r^rocess. iiowever, i ri suggesting specific 
criteria that might be u^ed, t:.e assumption is made that 
the future will require that: e^'aluation (1) should not be 
a process that excludes individuals from vocational educa- 
tion because of tneir specific weaknesses and (2) can no 
longer be a measure of quality of instruction. Quality in 
education has too long been confused with selectivity and 
tear of the conseque.ices of poor quality instruction has 
caused many persons to call for a return to guaranteed 
outcomes through careful selection of students into the 
program. This may be one of the reasons the federal govern- 
ment is now investing over ^10 billion in CET.a and 
only $600 TP.iliion in vocational education. 

nual i ty of Instruction 

Process Criteria for Instruct, on 

1. The deciree to v;hich alternati^^e methods are 
aval lable and used to meet individual student 
interests, aptitudes, and future working condi- 
tions. This suggests that a single methodology 
cannot be equally effective among the students. 

J. Th*.' degree to which time and organization are 
flexible to meet the needs of a variety of 
students . 

3. The clegr^ • to v;hich tool learning skills are 
of f ered and lea rned in orde r that the student 
may be a continuing learner on the job. 

4. The degree to which new and successful instruc- 
tional [:trocesses are rewarded as contrasted to 
tradit ional instruction . 

"j. Staff development related to instriicti .al 
priorities should be more than simply more 
education chosen by the individual staff member. 

6. The derjree that learner assessment and individualized 
procfram I'lanninq is in use. 

7. The degree to which outside advice from students, 
parents, employers, and citizens is used to 
redesign in struct ional processes. 

8. The amount of i. nd i vi "iua 1 i zed instruction offered. 
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Product Criteria in Instruction 

1. The degree to which all students achieve 

entry level comp'^tencies in an occupational area. 

2. The range of skill and learning beyond minimum 
competencies for every studenv. . 

3. The attitude of consumers, parents, students, 
and employers as to the quality of instruction. 

4. Measures of the degree to which vocational 
courses taken meet individv.al student career 
plans. 

5. The degree to which s^.udents leave prior co 
completion to secure employment and with minimum 
occupational competencies. Early learning may 
bo a criterion of success. 

6. The deqroe to which vocational instruction has 
created understanding by the student of the 
need for otner anu additional learning, 

7 The doqror^ to which vocational students have 
know: jdqe about and can demonstrate employ- 
ability skills. 

8. The degree to which students may select or be 
employed for *,ork requiring competencies beyond 
* he occupational study area. 

9. A measure of inc ividual career growth after 
leaving the proqram as compared to matched 
students without vocational educati- 

10. A measure of students* opinions of the value of 
vocational education some years later. 



Relevance of Program . This concept of account- 
abiirty~^rlt:erTa"would measure the quality of vocational 
education against t - criteria of judgments by those 
being taught and the actions of those who finance and 
use the product. In the long and sometimes short run, 
persons outside education select evaluative criteria 
and often this judgment is more perceptive than those 
whose vested interest that tends toward continuance of 
program and process and evaluation. These criteria also 
use the actions of potential students and the actions of 
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the vocational educator as specific indicators of 
the relevance o:' the Lroaram as related to individual 
needs and soci^'tal chanq»^s in the work area. 

Ktj 1 tj\ an CO c : P ro - ' jtv. 

Process Cr i ter i a 

1. Do all students secure related e -perience 
necessary for entry lob placement? 

2. Are processes established which assess student 
competencies up."^- entering vocational education? 

3. Have occupational changes in the area over the 
last few years resulted in program drops and 

chancres ? 

4. A: • special pr(3(irams, methods, and evaluation 
in uS'.- to assijt minorities, disadvantaged and 
hand i canned ? 

5. Do "power structure" persons ser 3n advisory 
committees? 

Are Students involved in the school program 
and assiqned responsibility as they are able? 

7. Are student youth ciroups an important part of 
the proqram. Are they delegated authority and 
respons i bi 1 i ty 

8. Are proqrams ijlanricd to inform students, 
narents, and employers about vocational 
educat ion ? 

Are outside people with special knowledge 
used in the instructional proqram? 

10. Do "f>ov;er structure' parents have their children 
enrol 1 - d .* 

11. Are students allowed to fail ana overcome their 
f ai lures ? 

1*^. Do students cn^jpCi jte and assist one anot- 

m th'' learnir nr_.r.es. and, if so, to wh^. ..egr- 



13. 



.students c\\rr\ or: self-evaluation? 



14. Are periodic reviows mado to determine programs 
that are not re levant ? Are they dropped? 

Product Criteria 

1. Do employers hire vocational students prior 
to hiring nonvocational applicants and to 
what degree? 

2. Do students learn employment seekinq skills 
and demonstrate these skil Is ? 

3. Are vocational staff involved in solving 
economic , manpower , poverty , and vocational 
related problems in the community? 

4. Do F-ti:iI<L:-' ts learn the latest knowledge regarding 
work; t'np.; oyment , advancement, and the re- 
quirrments of each? 

5. Do students feel more confident, self-motivated 
future oriented, and capable of their own self- 
direction as compared to nonvocational students? 

6. Do businesspeople and parents have more 
confidence in the vocational student succeeding 
in the work world? 

7. Do students choose the program and create 
demand or are efforts solely at recruitment? 

8. Do graduates secure additional education to 

a greater degree than nonvocational employees 
in the same setting? 

Impact of the Program . The concept behind this 
broad criteria is based on the fact that if something 
works or is needed most persons will buy it or to put 
it another way, changes will be made so the new, success 
ful, and valued program can grow. More money will be 
invested and schedules and priorities will change to 
allow the new program to try new things. In other words 
a success brings more success and support. Since 
vocational education functions in the environment of 
other education it is necessary to secure some change in 
that environment to be more effective in vocational 
education. Evidences of such successes are criterion 
which may be looked for as ways to evaluate the quality 
of vocational education. It is a type of consumer 
evaluation . 
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Process and Product 

1. Is the schedule and organization of the school 
or are regulations of the board of education 
changing to accommodate new programs and 
procedures ? 

Is the record of educational equity better 

as compared to other programs in the educational 

organ ization ? 

3. Are special funds offered to and used by the 
vocational department to try and test new 
things and new ways? 

4. Do other educators (nonvocat ional) visit the 
school to see new programs and processes? 

5. Do the media [people know about the vocational 
program and do they te 11 about it? 

6. Are vocational staff used as advisors in programs 
outside the school; in school? 

7. Have state regulations changed to encourage 
and reward new programs? 

8. Are federal regulations, laws, and appropria- 
tions for vocational and vocational related 
programs changing to accommodate the newer 
thrusts ? 

9. Are parents involved in the process of planning 
their children's education? Do they want to and 
do they feel it is valuable? 

10. Are services being expanded to meet the needs 
of adults at the request of the adults? 

11. Are vocational students respected as students 
and has the ratio of poor, dropout, and other 
isolated youth tended more toward normal 
distribution . 

12. Are employers calling on the school? 

13. Are youth vol untee.-'ing to serve the community 
as t^ar t of the p-'J^^^'-^^i^^rT^ ^'^^ aqer.cies and 
employers request the ir hexp? 
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14. Do adult attitudes toward ne" vocational 
programs, objectives, and pr ■-.•s^cs support 
what h^s happened? 

15. Are activities planned to impact upon the schools, 
parents, and community? 

16. Are new and different individuals and groups 
being served by vocational education? 



Individual Transition and Growth. The ultimate 
criteria for which vocational education should be held 
accountable is what happens to the learner when he/she 
makes the transition to the work world or to the next 
step whatever it may be. There are many factors other 
than vocational education which may determine the success 
of the transition but the ultimate question is what 
difference did vocational education make? The process 
of transition in our society has become a major problem 
for everyone as they move through the passages of life 
in a mobile, changing, complex society. 

If what happens after leaving the program is to 
be considered as one kind of criteria, then there must 
be some process by which the transition is made in such 
a way that the individual gets to use the vocational 
education he/she received and perhaps even more important 
that the individual moves into a situation where 
continued learning can take place. The goal of voca- 
tional education may be seen as not only preparing tor 
entry into the workplace but developing a learner whose 
qrowth in the work world will be self-actualized by the 
learning received in vocational education. Furthermore, 
the economic loss to society in unemployment is small 
compared to the loss due to under-employment which may 
have more long range consequences both to the individual 
and to society. These criteria see the job as a means 
not the end and thus raise the question of whether 
placement in a job for which trained is a good criterion 
against which to evaluate vocational education. It can 
certainly not be used if the average person changes 
occupations five times during his/her work life. 
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Process Criteria 

1. Is assistance available for transition to the 
work world? 

2. Is preparation for makincj the transition 
available and are minimum competencies es- 
tablished? 

3. Do parents approve of this service, do employers, 
do the students? 

4. Is there instruction to help students gain 
knowledge and skills to avoid the failures 
that 95 percent who lost their jobs have not 
learned ? 

5. Is follow-up of graduates used to change the 
program and process ? 

6. Do employers and community want to have input 
into educational criteria based on their involve- 
ment in transition and follow-up of students? 

Dc they have input? 

7. Are the community citizen and community agencies 
used in the transition process? 



Product Criteria 

1. Do students, em^jloyers, and community use the 

service ? 

2. Do students using this service secure work or 
employment with greater opportunities for utilizing 
their talents and training? Is a pattern of 
change in this use discernable? 

3. Do vocational students have long range career 

[)lans ? 

4. Is there evidence that vocational school learners 
do better at finding, receiving and advancing 

in the work world toward career goals? 

5. Do employer attitud.^s reflect a positive 
difference regarding vocational students as 
compared to nonvo::ational students in regards 

to ability to get, hold, and advance on the job? 
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6. Do vocational students have better knowledge 
of and understanding about future work and 
present workplace conditions, requirements? 

7. Are vocational students better informed as to 
future labor force trends and problems? 

3 . Are vocational students more knowledgeable 
regarding their own talents, interests as 
related to career plans than other students? 



Limitations 

This paper proposes some new criteria that may be 
more in the realm of "consumer" evaluation than "scientific 
measurement." Right or wrong, the matter of whether the 
quality of vocational education can be measured is a 
hypothetical question , but the matter of accountabi lity 
is a pragmatic, operational matter. It is assumed that 
researchers and evaluators as well as traditionalists 
in education all want to find ways to be more successfux. 
In a democracy, and even more in an educational system 
which has over 25,000 individual units from K-6 to 
separate vocational-technical colleges, each independent 
in terms of administration and operation, the determina- 
tion of success will be established by the owners and 
operators of the individual educational units. What is 
"right" is even more ethereal than what is "successful." 

There is a true and proper conflict between what 
ought to be and what is and one of the limitations of 
the criteria suggested in this paper is that too much 
emphasis may be placed on the pragmatic. However, the 
degree to which the citizen will support the "theoretically 
right" is often based on the perception of how well the 
researcher and evaluator may help solve and assess the 
solutions of the pragmatic, operational problem. 

No operator, separated from the theoretical, can 
help but become out of touch as does the researcher or 
evaluator isolated from the practitioners. 

In a sense, all of us in education are theorists 
as education has become isolated from society and 
especially in the work aspect. 

Most decisions about education are ''value judgments" 
and the long term basis for vocational education improve- 
ment will relate to value judgments rendered by users 
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of and participants in the program, as well as the 
public official who is accountable for financing and 
solving societal problems. 

The present direct relationship between education 
and work, now required for nearly everyone, requires a 
new look at evaluation irothods and priorities in 
educational research. This very fact may be a limitation 
of the proposed criteria, yet the question of what ought 
to be should not prevent analysis and assessment of 
what is. 

Another limitation of the proposed criteria is that 
they put overt and covert pressure on the practitioner 
who has had little if any preparation as to knov7ledge 
and skills required to accomplish the goals and the 
criteria may meet oppos it ion wi thin the educational 
community especially among non vocational colleagues, 
who may claim them to be irrelevant to "the true and 
proper" goals of education. 

Lastly, most researchers have been prepared to do 
a different type of analysis than called for by these 
criteria. The ability to advance professionally in the 
research and evaluation field is not dependent upon the 
kind of assessment suggested in this paper. 

Lastly, these criteria are begged, borrowed, and 
stolen and certainly some are wrong--that certainly is 
a liniitation. Some are new but only in context. 

Most of you reading this paper will be able to 
suggest other limitations. 



R ealistic vs. Ideal Criteria 

What is realistic and v/hat is ideal? In attempting 
to respond tc^ this often argued classification this paper 
simply defines realistic as criteria that are known, 
specific, accepted, and measurable. Idealistic criteria 
are those which are not commonly accepted as one's 
responsibility, dimly seen, and variably described, and 
for which there are few recognized measurement processes; 
however^ they are of such a nature that most persons 
believe "someone ought to do something about them." 

Ideal criteria can be used if they are put in 
understandable terms and if there is proc»f of positive 
help for the individual student , employer, and taxpayer. 
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Emp loyment in the Field for Which Trained 



This criterion, while appearing to be realistic 
and relevant, tends to keep vocational education tied to 
the past. The evidence is that such a criterion can only 
guarantee that vocational education will be seen as 
inefficient since there are too many other factors over 
which vocational education has no control that impact 
upon this situation. 

The factors that determine whether an individual 
student can get employment in a specific field are primarily 
those which are set outside education. At any one time the 
rate of change in skills, knowledge and technology is so 
great that employment in a specific field is more dependent 
on the following: 

1. local, state, and national economic conditions 

2. mobility of employer and employee 

3. government contract awards 

4. the shut down of production in a related field 

. discovery of a new energy source or a new machine 

6. the quirks of the interviewer 

7. the ^'eather 

8. the intornatic al conflict picture 

9. the wealth of the Arabs and the price of oil 

10. the changing values or life styles that are "in" 

11. others 

This is not to imply that failure to get work in a 
field for which the student is trained, if jobs are avail- 
able, is not a measure of [}Oor quality in vocational educa- 
tion, but rather it is to say that this criterion, when it 
dominates the measures of accountability, guarantees that 
vocational education would become less and less concerned 
with the future and work change and thereby less and less 
able to develop new criteria and programs responsive to 
changing conditions. 



The concept of employment in the field for which one 
is trained needs to be redefined to one that is mere relevant 
to today's workplace and to the individual's need to be 
able to adjust to the unknown changes in the workplace that 
will continue to come in the future. In fact, many 
employers argue that the greatest weakness of vocational 
education is adherence too closely to specificity of skills 
which can become obsolete or prevent occupational mobi 1 ity 
either vertically or horizontally. 



C onclusions 

Through this paper , the author has attempted no more 
than to suggest that traditional criteria for accountability 
in education and in vocational education are not relevant 
to the future. Measurement of outcomes against only 
traditional and validated criteria assigns vocational educa- 
tion to a decreasing role in preparing individuals for future 
work at a time when preparing for work and changes in work 
roles are necessary for everyone. 

The new and specific needs of special groups and the 
new and continuing needs in education apply to everyone 
today and not just to a percentage of the population. 
Another way of expressing the arguments for the con- 
clusions reached by this writer is that success criteria 
in both work preparation and in general education which 
have been traditionally used are not adequate. Somehow, 
the two must be put together and changed if vocational 
education is to meet the needs of both learner and society: 
Such redesign will require changes in product, process and 
in institutions which are responsible for product and process. 

Therefore, the following conclusions should be given 
consideration in selecting criteria for accountability. 

1. Quality in vocational education cannot be tied to 
the old normative measures designed to select 
students out of formal education or to select 
workers to proceed into the higher skilled oc- 
cupations . 

2. Criteria selected must be relevant to the individual 
needs as well as to new societal problems di fferent 
from the past. 

3. Vocational education methods and processes need to 
focus more directly on the re lat ion ships and 
learning opportunities in society that wi 11 bring 
education and work experience together in a 
cooperative effort. 
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4. Search for specific criteria that can be proven 

" to be ric7h t " rr.ay prevent the search for criteria 
that nay bo more usable ideal i st ically and real is- 
tical ly . 

5. Criteria for accountability must bo selected that 
are responsive to student needs, societal problems, 
and employer concerns before the evaluation processe 
can be determined. 



S[^eci 1 i c Criteria Suggestions 

This paper has made no attempt to be definitive in 
the presentation of specific criteria other than the sugges- 
tions made earlier. Perhaps if the suggested broad areas 
for criteria of accountability make sense a national effort 
to define specific criteria by a group made up of employers, 
parents, students, teachers, administrators, policy makers, 
and legislators could be chosen to define specific criterion 
in each broad area. This could be a basis for redesigning 
the criteria for accountability against which vocational 
education should be held accountable. 

This effort ouqht to be made by those in vocational 
education so the concern for quality by vocational educators 
is obvious to those who now see compliance and regulation as 
the route to forcinu vocational education to move tow rd new 
criteria. 

The four broad cacu*qorios of criteria to be studied 
are the followintr: 

1. Instructional ar.d [.-rograra quality, 

2. Proqram relevance' to individual and societal needs 
in relation tcj work, 

3. Proqram impact on organization, policy, support 
and usacfo of vocational educatiori, 

4. Individual transition to and growth in the work 
wo rid. 

Unless the vocational c^ducator and the evaluator of 
vocat ional educat ion } roposo new and better criteria 
acceptable to society, oven more time will be spent in 
meeting compliance requi ror^cntis which are becoming ends 
instead of means. 



Therefore, this writer [)roposes t-hat those four broad 
categories of criteria boci/fne the basis for establishinq 
specific criterion m each area that can assess the broad 
changes of vocational education in specific units educating 
students or administering procrams. 
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REACTION TO CRITERIA AGAINST WHICH VOCATIONAL 
EDUCATION SHOU'^D BE HELD ACCOUNTABLE* 

We are indebted to Dr. Venn for his insightful and 
stimulating treatment of a most difficult question with 
which this conference is f^roperly concerned. It is 
proper that we are conceri.od with criteria and outcomes 
because they are at the very heart of the enterprise. What 
we choose as criteria for assessment , what we agree to 
accept as evidence of success, defines our purpose. We 
cannot divorce our choice of criteria from our concept of 
the proper role of vocational education. No matter what 
philosophical statements are made, no matter what goals may 
be stated in general terms, the criteria we accept will 
reveal our real (working) concept of vocational educa- 
tion — the one that matters. 

Although it probably is wise to exclude *'How to Measure" 
considerations from initial work on the selection of 
criteria, as suggested by Dr. Venn, it is important to 
remember that our definitions are incomplete until our 
measurement operations are defined. In fact, we are not 
dealinq with two different matters--criteria and how to 
measure them--we are talking about a two-step process of 
defininq criteria: (1) say it approximately, (2) say it 
more precisely. 

Even though we minht agree on the importance of criteria 
and their clear definition, there remain two problems in 
arriving at a set of criteria against which vocational 
education should be h-^id accountable. First, note that Dr. 
Venn says, . . . regardless of how well vocational 
education can measure and prove that it is meeting its 
objectives . . . many persons question the objectives 
or criteria in use." This is another way of saying that the 
choice of criteria is inevitably a value question. There 
is no value-free analysis by which to select criterion 
outcomes. Since different stakeholders have different needs, 
different goals, and different priorities for reducing 
problems- they also have differing exp^ctations concerning 
the outcomes of vocational education. 



* Reaction to Grant Venn's paper by Edward J. Morrison. 



The second problem that shows up (as Dr. Venn often 
points out in his paper) is that the social, economic, 
psychological, working world is complex, changing, even 
unknowable • 

All this makes for a pretty hopeless picture. Goals 
are essential and become practical when measurable, but 
there is no consensus available on goals because stake- 
holders differ and because the world is changing kalei- 
doscopically in ways which cause the stakeholders to 
change their goals continually-- thus to increasing the 
chaos . In such circumstances , lists of specific criterion 
measures '^re not sufficient — they change too soon and we 
must begin again and again . We need a conceptual device 
for coping with the uncertain future . 

It may be that Dr. Venn has subtly suggested a device 
which is more f ami liar in the fields of measurement and 
philosophy of science — the notion of construct validity. 
Perhaps we can identify a few criteria about which we could 
say at any particular time, "If vocational education meets 
these criteria, the following specific things would be 
likely." Such criteria would rarely change, but the indica- 
tors would be adjusted as necessary to track changes in 
Dr. Venn's changing world. Relevance of program might be 
an example — a construct. If we could agree that vocational 
programs always should have relevance for the needs of 
students and of other elements of society, a number of 
indicators of relevance could be chosen at any time from 
measures appropriate for the times. In this way, the 
specific indicators might change, but the basic criteria 
would remain the same. 



-196- 



2^: ^ 



REACTION TO CRITERIA AGAINST WHICH VOCATIONAL 
EDUCATION SHOULD BE HELD ACCOUNTABLE* 



First, I coiTunend Dr. Venn for his provocative and 
nontradi t- ional ideas about the criteria against which 
vocati education should be he Id accountable . The 

pre^ n and presentation of a paper on evaluation in 

voc ■ education that barely gives a glance and nod 

in tiiu . rection of federal legislation and its accompanying 
rules and regulations is a feat worthy of note. I have 
little doubt but that the implementation of the ideas he 
proposes will relieve us to a considerable extent from what 
was re f erred to yesterday as the " tyranny of the criterion 
of placement" as the accountability measure of vocational 
education. On the other hand, the application of his ideas 
would probably be accompanied by considerable stress and 
difficulty on the part of vocational educators and the 
various audiences of vocational education. 

In reacting to the paper, I will attempt to extrapolate 
what I see as some major ideas or concepts regarding 
the evaluation of vocational education that merit attention, 
debate , re f inement , and --when war ranted-'-systematic 
testing and appraisal. I believe Dr. Venn is calling our 
attention to some important issues in accountability and 
evaluation that we can ill afford to ignore. 

As I comni- t further, I want to make it clear that, in 
effect, I am "interpreting" since that is a word in the 
title of the project of which this conference is a part 
that may not be receiving the attention it merits. The 
project concerns the interpretation of outcome measures 
in vocational education. In a real sense, there are a 
number of factors that influence the interpretations 
persons or qroups place on data and information. One of the 
factors havincj a major influence is the values held by 
those who interpret. I want to comment further on the 
idea of interpretation later in these remarks. 



Reaction by J. Robert Warmbrod to Grant Venn's 
presentation . 
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Let me indicate and comment briefly on what I see 
as five major ideas concerning evaluation in vocational 
education discussed or alluded to in the paper. 

Outcomes of vocational education cannot be interpreted 
validly in i sol at ion from process . The underlying rationale 
of outcomes as accountability criteria requires a functional 
relationship between process — instruction — and outcomes. 
Data describing purported outcomes , however valid , have 
limited use for policy and program decision making in the 
absence of a description of the process that supposedly 
had something to do with producing the outcomes observed. 
I suggest that the energies devoted to de fining a "voca- 
tional program" might be better invested in developina 
a system that facilitates the accurate description 
multitude of different processes that those enrol ■ 
vocational education experience. Perhaps our qoa e 
to describe the diversity of vocational programs r. -in 
assuming that there is "a vocational program." 

A second big idea concerning evaluation that I believe 
we should pay attention to is the appropriateness of 
employment as an outcome criteria . Dr . Venn ' s position is 
quite clear — transition to the world of work, employment, 
and advancement are multi-factor phenomena; vocational 
education has little if any control over many of the 
factors involved. He puts it this way: "There are many 
factors other than vocational education which may determine 
the success of the transition but the ultimate question is 
what difference did vocational education make?" 

Perhaps I can illustrate this point with a "Dear Abby" 
column that appeared a few years ago. A young girl 
writes : 



Dear Abby — I have a problem that may be of concern 
to other girls my age. I have just begun to date 
and it seems that if I do not let a boy kiss me 
on the first date I never see him again. But 
if I do let him kiss me on the first date I 
never see him again either. So how is a girl 
supposed to know what to do? Just Wondering. 

Dear Just — First you can conclude that whether 
or not you let a boy kiss you on the first date 
has nothing to do with whether you see him again. 
I suggest you go to work on finding other poss ible 
causes of you being a one-date Dolly. 
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If we look at the major evaluation studies of the out- 
comes of vocational education, we tend to find ourselves 
in somewhat the same dilemma — there is little difference in 
outcomes between those' who complete vocational education 
and those who do not. We need to be aware that the outcomes 
frequently measured have "other possible causes" with some 
of these causes having a more direct and pervasive impact 
on the outcomes measured than does vocational education. 
This issue continues to surface during this conference 
under the terms employabi 1 i ty versus employment . 

Another concern alluded to in the paper has to do with 
the relationship between criteria and measurement. Although 
Dr. Venn does not discuss this connection at length, the 
gist of his remarks seems to be that close at tent ion 
measurement problems and concerns may impede or ' . 
the generation and formulat . f appropriate \X \ 

criteria. Actually, the gene tion of criter^^a .aid 
measurement are not separate and distinct but stages in the 
conceptualization and operational development of outcome 
measures . 

Dr. Venn also reminds us that to evaluate is to judge. 
He states "Most decisions about education are *value 
judgments* and the long term basis for vocational education 
improvement will relate to value judgments rendered by users 
of and participants in the program, as well as the public 
official who is accountable for financing and solving 
societal problems." This brings me back to my earlier 
comments about interpreting outcomes. If valid interpreta- 
tions of outcome data are to be made, a first essential 
is that the data be "true," that is, valid. It is rather 
obvious that we have ample opportunity for improvement on 
this score as far as present outcome data are concerned. 
Actually, this is primarily a measurement problem issue. 
Another dimension of the truthfulness — the believability — 
of outcome data has to do with the extent to which factors 
other than vocational education may be influencing the 
outcomes observed. These constraints must not only be 
recognized but their probable impact on strengthening or 
qualifying interpretations needs to be assessed and made 
evident. But even with valid and reliable data, that all 
will accept as fact, it is not uncommon for varying 
interpretations to result, at least in part, as a con- 
sequence of the values or poi nt-of -view held by those 
who judge. 

The last major point I want to draw from the paper is 
that criteria against which vocational education should 
be held accountable cannot be separated from the purposes , 
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goals, or objectives of vocational education. From this 
presentation we have a rather clear indication of what Grant 
Venn sees as important purposes of vocational education. 
The degree of agreement within the profession on appropriate 
evaluative criteria and agreement between the profession 
and its various audiences on accountability criteria will 
not exceed the degree of agreement about the expectations 
W3 hold for vocational education. In essence, those who 
set accountability criteria also determine goals and 
objectives . 

This leads to a concluding comment , prompted not only 
by Dr. Venn's paper but also by other comments voiced 
during the past two days — perhaps most directly by Mr . 
Jennings on Wednesday evening. The criteria against which 
vocational education should be held accountable are 
specified by the claims we make for vocational education 
Tf t-^ laims .re grandiose, encompassing all emp' ' , 

per.-, jnal \nd so> lal concerns over which the school id 
less vocational education has little if any control , 
then we should not be too alarmed when the assessment does 
not substantiate vocational education as the solution to 
all ills. In effect, we get what we ask for; and some of 
our perceived antagonists would probably add that we get 
what we deserve. 
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ANALYSIS OF EXTANT DATA BASES 
IN VOCATIONAL EDUCATION* 

Background and Need 

The placement rates of students as an outcome of^ 
vocational education programs have been a source of increasing 
debate. On the one hand, those who view vocational educa- 
tion as a focused and targeted system of training for 
specific occupations indicate placement rates are the 
only "true" measure of program effectiveness. On the other 
hand, those who view vocational education in a broader 
concoxt indicate that Lhere are other outcomes that should 
be considered when evaluating vocational education. 
Regardless of these two divergent points of view, the 
Education Amendments of 1976 require each state to: 

. . . evaluate the effectiveness of each program 
within the states being assisted with funds avail- 
able under this Act — by using data collected, when- 
ever possible, by statistically valid sampling 
techniques, each such program within the state 
which purports to impact entry level job skills 
according to the extent to which program com- 
pleters and leavers (i) find employment in oc- 
cupations related to their training, and (ii) 
are considered by their employers to be well 
trained and prepared for employment . . . 

Similarly, earlier legislation (e.g., Vocational 
Education Amendments of 1968, and the Vocational Education 
Act of 1963) also required states to conduct follow-up 
studies of vocational students. As a result of the 
legislation cited above, a number of follow-up studies 
have been collected and each state has reported a placement 
rate for the state. These placement rates by states, 
as well as other vocational education data, have been 
reported by Lee and Sartin (1973). However, Lee and 
Sartin (1973, p. 15) prefaced their findings by stating: 



Presentation by N. L. McCaslin. 
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. . . the data (obtained from states) are neither 
uniform nor complete. In many cases, they are 
obviously inaccurate, and can easily be misleading. 
To use them at all raises a question about the value 
of summaries, analyses, and conclusions based on this 
kind of material. 

Lee and Sartin (1973, p. 15) went on to state that: 

There are numerous problems with the da*:a being 
collected by state and federal agencies which should 
be recognized , and these problems should be dealt with 
if the government's responsibility for public funds 
is carried out. They fall into three groups: problems 
of definitions; problems of communication; and problems 
of careless or dishonest handling of data. 

Therefore, if 'me -l ita are to be inij^Lovca so thai 

better interpretation can be made, one area in which systematic 
investigations will be needed is to demonstrate what happens 
to placement rates as definitions of relatedness are changed. 
These investigations could then be used to illustrate the 
effects of differing definitions of placement in occupa- 
tions related to training, to personnel responsible for 
follow-up studies of vocational education completers and 
leavers . 



Types of Information Needed 

Prior to discussing how these studies might be conducted, 
it is necessary to indicate the minimum types of information 
that would be required to carry out the investigation. 
This study could be conducted in a limited manner using 
different definitions of relatedness if the following student 
information were available: 

1. data on the vocational program in which a student 
^ is enrolled coded by USOE program codes, 

2. data on current occupation coded by the number of 
the DOT job title, 

3. teacher or student judgment of relatedness (e.g., 
related, somewhat related, unrelated) , and 

4. a computerization of the information included in 
Matching Occupational Classification to Vocational 
Education Program Codes (U. S. Department of 
Labor, 1975) . 
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The remainder of this section will present a brief review 
of each of the four requiren^ents listed above. 



The USOE Program Codes were designed to help local and 
state educational agencies identify, classify, and describe 
information about subject matter and curriculum. In 
vocational education, the codes describe similar groupings 
of subject matter rather than well-defined courses. 
Additionally, the programs are not limited to any educational 
level (e.g., elementary, secondary, or pos tsecondary ) 
and are applicable to several levels. 

In vocational education the codes are largely limited 
to a six digit number. These r'^i'Ti^" are foll^-'-- 

. .j i JuL jcct-mat ter area : 

01 . Agriculture 

04 . Di stribut ive Education 

07 . Health Occupat ions Education 

09. Home Economics 

14. Office Occupations 

16. Technical Education 

17. Trade and Industrial Occupations 

Second 2 digits Principal segment of subject matter 

Third 2 digits Division of principal segment 

A comprehensive list and description of these programs are 
included in Standard Terminology for Curriculum and 
Instruction in Local and State School Systems (U. S. 
Department of Health , Education, and Welfare, National 
Center for Educational Statistics, 1970). 

DOT Job Titles are basically numbered using a nine- 
digit code. The first six-digits reflect the kind and level 
of work performed, while the last three digits are assigned 
to further differentiate occupations. The Dictionary of 
Occupational Titles (U. S. Department of Labor, 1977) 
presents a detailed explanation of the classification 
structure . 

Teacher or student judgment of relatedness is generally 
in the form of related, somewhat related, or unrelated. 
In making this decision individuals are asked to make a 
judgment on the degree to which their training is related 
to their present occupation. Some states, such as 
Oklahoma, ask students to indicate whether: skills 
learned in vocational education are highly significant 
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in this occupation, skills learned in vocational education 
are significant in this occupation, or skills learned in 
vocational education are not significant in this occupa- 
tion. 

Matching occupational classifications to vocational 
education program codes has also been discussed in a publica 
tion of this same title (Department of Labor 1975). This 
publication matches vocational education program codes with 
related Bureau of the Census codes and Dictionary of 
Occupational Titles codes. tbii unat or • 

rnmr : -rr^ V m 1 t hf. > i^G asked to cl*.^Lci.iuxne 

, I , . occupation were related to his or her 

training and the placement rate could be determined for 
given vocational programs - 



Proposed Procedure for Comparing Placement Rates 
for Differin g Definitions of Relatedness 

If a data Dase had information on all four of the items 
previously discussed it would then be possible to demon- 
strate the effects of the different definitions of related- 
ness on placement rates. The purpose of this section is 
to illustrate how the comparison could be accomplished. 
In conducting this analysis, the following four major steps 
would be needed : 

1. Determine the USOE program codes (to the sixth 
digit) included in the data base. 

2. Determine the number of students in related oc- 
cupations, using Matching Occupational Classif ica- 
tions to Vocational Education Program Codes , 
(DeparErient of Labor 1975) , for each of the three 
successive combinations of two-digit numbers of the 
USOE program codes included in the data base. 

(If the information were computerized so that the 
information on vocational program codes and DOT 
job titles could be crosswalked, this would 
be fairly routine.) 

3. Compute percent placement rates for each definition 
of the three vocational education program codes . 

4. Compare these percentages with those obtained 
using the teacher or student judgment of related- 
ness. 
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The following section will discuss how this could 
done operationally for a secretarial program in vocati 
education. If a secretarial program in vocational ec'' 
tion is defined as 14 . 0000 (Office Occupat ioi:s) then •> i 
occupations (accor^Tinq to the Department of abor pubi: 
i on r>»-ri above) : ude the follow in d ti^lc.: 





- • d-010 


. ..iLcher, motor vehicle 




. -.8-024 


Survey worker 


249 , 


. 368-010 


Claims clerk 


24 9 , 


. 368-034 


Credit re port e r 


160 , 


. 288-018 


Es t ima to r 


221 . 


. 168-010 


Material coord i na tor 


221. 


168-022 


Product ion coordinator 


221 . 


388-054 


Production clerk II 


222 . 


368-014 


Expediter I 


206 . 


388-018 


File clerk 


205. 


368-026 


Personnel clerk 


168 . 


288-014 


Claim examine r I 


249 . 


368-0 50 


Library assist an t 


233 . 


388-010 


Mail carrier 


228 . 


388-014 


Rural mai 1 carrier 


231 . 


588-015 


Mail clerk 


239 . 


587-010 


Mai le r 


230 . 


878-022 


Messenge r 


230 . 


878-026 


Office he Iper 


24 9 . 


388-038 


Messenoer. bank 


239 . 


588-018 


Meter reader 


215 . 


488-010 


Pav ro 11 cl G rk 


219 . 


388-274 


Timekeeper 


232 . 


368-010 


Dist ribu t ion clerk 


209 . 


688-034 


Proofreader I 


209 . 


688-014 


Insurance cle rk 


237 . 


6 38-033 


Recept ion is t 


222 . 


587-042 


Shipping clerk 


222 . 


387-026 


Shippinq and receiving clerk 


222 . 


387-018 


Rece i V incj c le rk 


219 . 


388-258 


Statistical clerk 


219. 


388-074 


Coding clerk 


219 . 


488-034 


Cost clerk 


249 . 


368-026 


Court clerk 


223 . 


388-014 


Inventory clerk 


230 . 


868-010 


Telegraph messenger I 


236 . 


588-014 


Telegrapher 


235. 


862-018 


Information operator 


203 . 


588-014 


Telegraphic typewriter operator 


235 . 


862-026 


Telephone operator 


235. 


862-010 


Central office operator 


235 . 


138-010 


Central office operator supervisor 


235. 


138-014 


Telephone operator chief 
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21 J 

o 

ERIC 



249 . 


368- 


030 


Credit clerk I 


120 . 


188- 


026 


Programmer, busine.- 


012. 


168- 


022 


Systems analyst, bu us electronic 
data processing 


219. 


488- 


010 


Accounting clerk 


249 . 


368- 


050 


Library assistant 


161. 


268- 


010 


Clerical technician 


166. 


268- 


014 


Employment interviewer 


166. 


118- 


022 


Manager, personnel 


168. 


168- 


050 


Manager, credit and collection 


161. 


118- 


018 


Treasurer 


169. 


168- 


014 


Administrative assistant 


232 . 


138- 


010 


Supervisor, mail 


169. 


168- 


062 


Manager, office 


201. 


368- 


010 


Legal secretary 


201. 


268- 


018 


Secretary 


201. 


368- 


014 


Medical secretary 


202. 


388- 


014 


Stenographer 


202. 


388- 


010 


Court reporter 


209 . 


388- 


022 


Clerk typist 


203 . 


588- 


018 


Typist 


217. 


388- 


010 


Proof machine operator 


214. 


488- 


010 


Payroll clerk 


217. 


388- 


010 


Transit clerk 


219. 


388- 


026 


Billing clerk II 


216. 


488- 


018 


Calculating machine operator 


216. 


488- 


010 


Adding machine operator 


213. 


382- 


018 


Digital computer operator 


207. 


885- 


010 


Duplicating machine operator 


207. 


782- 


014 


Duplicating machine operator II 


207. 


782- 


026 


Offset-duplicating machine operator 


213. 


382- 


018 


Digital computer operators 


213. 


138- 


010 


Supervisor, computer operations 


213. 


782 - 


010 


Tabulating machine operator 


222. 


687- 


034 


Sorter 


209. 


698- 


045 


Sorter 


216. 


488- 


018 


Calculating machine operator 


234. 


582- 


010 


Addressing machine operator 


208. 


885- 


010 


Collator operator 


219. 


388- 


254 


Sorting clerk 


212. 


368- 


010 


Teller 


219. 


388- 


026 


Billing clerk II 


209. 


588- 


018 


Clerk, general 


210. 


388- 


210 


Bookkeeper 


211. 


368- 


010 


Cashier 


209. 


588- 


018 


Clerk, general 


169. 


168- 


026 


Chief clerk II 


219. 


388- 


066 


Clerk, general office 


213. 


138- 


010 


Supervisor, computer operations 


240. 


388- 


010 


Collection clerk 
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However, if a secretarial program in vocational 
education is defined as 14.07 (stenographic, secretarial, 
and related occupations) then related occupations include 
only the following five different job titles: 



Finally, if secretarial programs in vocational education 
are defined as 14.7092 (secretaries) then related occupations 
include only the following three job titles: 



Therefore, three different tables of data would be de- 
veloped for each of these three different definitions of 
secretarial programs. Table 1 illustrates how the number 
of students in related occupations would be displayed if 
related occupations for secretarial programs (14.0702) 
are defined as those appropriate for office occupations 
(14.0000). This table would have the 93 occupations that 
are considered related by Matching Occupational Classif ica- 
tions to Vocational Education Program Codes (Department 
of Labor 1975) . 

Table 2 illustrates how the number of students in 
related occupations would be displayed if related occupations 
for secretarial programs (14.0702) are defined as stenographic, 
secretarial and related occupations (14.07). This table would 
have the five occupations that are considered related by 
Mat ching Occupational Classifications to Vocational 
Education Program Codes (Department of Labor 1975) . 

Table 3 illustrates how the number of students in 
related occupations would be displayed if related occupations 
for secretarial programs (14.0702) were defined strictly 
as secretarial proarams (14.0702). This table would have 
three occupations that are considered related by Matching 
Occu pational Classifications to Vocational Education 
P rogram Code s (Department of Labor 1975) . 

Once the total number of former vocational education 
students placed in related occupations has been determined 
for each of the three classification systems previously 
discussed, a percent placed in related occupations could 



201. 368-010 
201. 368-018 
201. 368-014 
202. 388-014 
202. 388-010 



Legal secretary 
Secretary 
Medical secretary 
Stenographer 
Court reporter 



201. 368-010 
201. 368-018 
201. 368-014 



Leaal secretary 
Secretary 
Medical secretary 
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Table 1 



Suggested Table for Displaying Numbers of Former 
Vocational Students in Occupations Considered 

Related to Secretarial Programs (14.0702) 
When Relatedness is Defined as Any Occupation 
Related to Office Occupations (14.0000) 



1. Legal secretary 
201. 368-010 

2 . Secretary 
201. 368-018 

3. Medical secretary 
201. 368-014 

4 . Stenographer 
202. 368-018 

5. Court reporter 
202. 389-010 

6. Programmer, business 
020. 188-026 

7 . System analyst , business 
electronic data processing 
012. 168-022 

8 . Accounting clerk 
219.488-010 

9. Typist 
203. 588-018 

10. Teller 
212. 368-010 

11. Cashier I 
211. 368-010 



Related Occupations 
DOT Job Title Number 



Number of Former 
Vocational Students 
Employed in the 
Occupation 



12. 



94. 



Total 
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213 



Table 2 



Suggested Table for Displaying Numbers of Former 
Vocational Students in Occupations Considered 

Related to Secretarial Programs (14. 0 702) 
When Relatedness is Defined as Any Occupation 
Related to Stenographic, Secretarial, 
and Related Occupations (14.07) 



Related Occupations 
DOT Job Title Number 



Number of Forme r 
Vocational Students 
Employed in the 
Occupation 



1. Legal secretary 
201. 368-010 

2. Secretary 
201. 368-018 

3. Medical secretary 
201. 368-014 

4. Stenographer 
202. 368-018 

5. Court reporter 
202. 389-010 

Total 
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Table 3 



Suggested Table for Displaying Numbers of Former 
Vocational Students in Occupations Considered 
Related to Secretarial Programs (14.0702) 
When Relatedness is Defined as Any 
Occupation Related to Secretarial (14.0702) 



Related Occupations 
DOT Job Title Number 



Number of Former 
Vocational Students 
Employed in the 
Occupation 



1. Legal secretary 
201. 368-010 

2. Secretary 
201. 368-018 

3. Medical secretary 
201. 368-014 



Total 
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be computed by dividirui the total number of students in 
related occupations by the number of students completing 
or leaving the secretarial program. These percentages 
could then be compared with the percent placed in related 
areas using teacher or student judgment. Table 4 illustrates 
how this table would appear. 

Assuming the previous statements related to the types of 
information needed and the procedures for conducting a study 
are correct, then the next logical step would be to identify 
existing data bases that would be accessible for this type 
of analysis. 



Search Cor Extant Data Bases 

In searching for extant data bases, the project advisory 
council sugcjostod that the i^roject staff limit their search 
to state and larger local education agencies. These 
agencies were identified through telephone contacts to 
state and local education agencies thought to have the 
potential for this type of analysis and recommended by the 
project advisory council. In turn, the project staff was 
referred in some cases to other agencies who, in the opinion 
of the person contacted, might have the required types of 
information in an existing data base. 

The following education agencies were contacted. 

State Agencies Local Agencies 

Colorado Cleveland, Ohio 

Florida Gainesvi 1 le , Florida 

Kansas Mesa, Arizona 

Minnesota New York City, New York 

Missouri Oakland, California 

Ok lahoma 
Texas 

Additionally, the project staff carefully reviewed 
Occupational Information Resources : A Catalog of Data 
Bases and Classification Schemes (Ashley 1977) . This 
publ ication lists and describes a total of 55 different 
occupational data bases. 

As a result of the effort described above, no data 
bases were identified that had all the information 
identified as necessary for the study described for the 
project. Therefore, a case study of states known to have 
diffcrinq nlacomcnt rates was recommended to idr^ntify 
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Table 4 



Percent Placed in Related Occupations When Different 
Definitions of Vocational Programs are Used 



USOE Program Codes 



Program Title First 2 Digits First 4 Digits First 6 Digits Teacher and 

(14.0000) (14.07) (14.0702) Student Judgment 



Secretarial 



ERIC 



reasons for such wide discrepancies in placement rates as 
those reported by Lee and Fitzqerald (1975, p. 73) in which 
they reported placement rates for all vocational education 
students then employed (1974). The ratio of "employed in 
field other than that for which trained" in their report 
ranged from about 20:1 to 1:1.5. This type of range leaves 
little doubt that there is a need to try and identify why 
these rates vary so dramatically and why a case study of the 
phenomenon would be appropriate prior to designing empirical 
studies related to these "unknown factors." 



Summary 

Th i s paper has described the process that was followed 
in attempting to identify extant data bases for an empirical 
study of the effect of different definitions on placement 
rates of vocational education students. In this process, 
the need for the investigation, the types of information 
needed, the proposed procedure for comparing placement rates, 
and the search for extant data bases have been summarized. 

In conducting this search for extant data bases with 
which to investigate the effects of differing definitions 
on placement rates, a number of problems seemed to pervade 
this activity. For purpor-es of this summary, these problems 
have been grouped in the following three major areas: 
the definition of terms, the design of the evaluation, 
and the door of the classroom. 

The definition of terms seemed to be a major obstacle. 
In most of the data tapes students were classified as either . 
vocational, aeneral or college preparatory. Generally, 
students were not coded by USOE program codes and if they 
were, they usually included only the first two digits. 
No information was included relative to the length of time 
students spent in vocational classes. Current occupations 
were generally not included but v/ere reported as related, 
somewhat related, or not related. 

The design of the evaluation was the second major problem. 
Most of the studies identified were conducted tor oth^r 
purposes. As such, it was unfeasible to expect thar the 
data could be manipulated using yet another design. 

The th ird and final problem in dealing with extant data 
bases was the door of the classroom. Who is to know the 
actual content of the class reported by a USOE code ? 
Certainly , many deviat ions could be expected due to the 
varying situations one could expect, e.g., teacher dif- 
ferences, studuMit differences, geographic difference. 



etc. The reliance on teacher reported data from each 
classroom further complicates the situation • Althouqh 
teachers would be expected to report their figures honestly, 
certain variations would normally be anticipated due to the 
varying interpretations made from the original questions. 

Although the original intent to idcnt i f y existing data 
bases for secondary analysis was sensible, it is Siahly 
unlikely that this type of information is available. 
Perhaps vocational educators should consider joining 
efforts with other longitudinal studies and request 
data coded in a manner that would then be used in a study 
such as this paper described. Unt i 1 the information becomes 
aval 1 able , the impact that differing de fi nit ions of re la ted ~ 
ness has on placement rates will remain a question for further 
debate . 
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SUMMARY REPORT OF SMALL GROUP INTERACTIONS* 



It is difficult to summarize the small group inter- 
actions. However, it is easy to summarize the process. 
The small groups were very well organized and conducted. 
A diversity of professional backgrounds was represented 
as evidenced by managers, teacher educators, and researchers 
in each group. 

It is more difficult to summarize the content of the 
small group activity. TVnyone familiar with the topic of 
this conference is well aware that many dilemmas confront 
one when considering outcome measures for vocational 
education. These dilemmas were referred to by the 
speakers and confirmed in the small groups. 

I have chosen to summar i ze the report of smal 1 group 
interactions by listing the dilemmas. 



DIMENSIONS OF IDENTIFYING OUTCOME 
MEASURES FOR VOCATIONAL EDUCATION 

1. Sensitive versus sensible measures. 

2. Comprehensive versus operational definitions. 

3. Data for federally versus nonfederally funded 
vocational education programs . 

4. Federal versus state versus local. 

5. Education versus training. 

6. Loncf-range versus short-range benefit effects. 

7 . Respons iveness to cons ti tuency versus changing the 
va lues of our cons ti tuency . 

8. Best or least bad alternative for student. 

9 . Compliance versus improve decision making . 



* Presentation by George Copa. 
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10. Define use of data versus create need for data. 

11. Mandating process and reportina versus mandatincj 
qual ity . 

12. Formal versus informal information systems. 

13. Adversary versus advocate. 

14. Promise nothing versus promise evcryLhing. 

15. Standardized data bank versus specific studies, 
evaluation versus evaluative research., description 
versus relationships (cause and effects). 

16. Changing versus stable information demands. 

17. Consumer versus producer in defining purpose and 
outcome measures . 

18. We versus they--isolation of vocational education 
from CETA and other programs. 

19. Secondary versus pos tsecondary versus adult. 

20. Collecting versus interpreting. 

21. Let's get job done versus let's talk some more. 

22. States as same or dif ferent--percent of vocational 
education funds, expected placement rates, use of 
federal funds. 

23. Vocational education versus service areas. 

24. One way versus two-way flow of data. 

25. Simplistic versus complex. 

Perhaps this is over emphasizing the differences, but 
to have stressed the commonalities would have required 
much more time. There appears to be a frustration with the 
lack of progress to resolve the dilemmas. 

One suggestion is that vocational education needs 
a mini-congress within which to debate the issues, present 
evidence and to make decisions. Such a policy forum could 
contribute greatly to resolving some of the key issues in 
vocational education , 



PERCEPTIONS OF CONFERENCE ANALYST* 



The National Conference on Outcome Measures for 
Vocational Education is now history. There have been 
extensive opportunities for individuals to listen, think, 
and interact with one another on this timaly and important 
topic. In addition, participants have in their possession 
a notebook complete with the significant artifacts (e.g. , 
papers and con fere nee notes) for re fere nee and sharing . 
I want each of you to think creatively about how the 
information you have received might be shared within your 
own organization. Perhaps this material might even make your 
job easier if the appropriate individuals are made aware 
of even a small portion of the information you have received . 

The presentations were scholarly and delivered in an 
articulate and professional manner. Each individual v/as 
keenly aware of the audience to which he/she was addressing 
and knowledgeable of the types of problems the workshop 
participants were facing. Bob Morgan provided us with a 
report on the status of the Vocational Education Data 
System of which he is so knowledgeable. His discussion 
gave us an excellent updating of his most difficult job. 

The banquet speaker, Jack Jennings, gave us new 
insights into the Congressional perspective and their 
subsequent influence on federal legislation. The three 
major papers represent significant contributions to the 
field. Mary Kievit causes us to rethink potential outcomes 
by considering how they might be more closely related to 
the values of our various audi'^nces and provided a beginning 
for how we might deal with linking various values to programs 
of vocational education . Donald Drewes asked us to think 
about the use of the evaluative information to improve 
vocational programs. His proposal asked us to consider 
a feedback link from the federal to the state and associated 
agencies. Grant Venn's provocative paper has asked us to 
consider that traditional outcome criteria are not relevant 
for the future and that we begin to suggest new and better 
criteria, acceptable to society so that less time will 
be spent in meeting the requirements of the legislated 
and subsequently compliance efforts in evaluation. 



Presentation by N. L. McCaslin. 



The reactors to the three papers were perceptive and 
caused us to reflect upon each topic from a slightly 
different perspective. 

George Copa has assembled information on concerns and 
comments previously discussed in the small groups. 
However , I might also state that the smal 1 groups provided 
dialogue that was necessary to maximize the benefits of 
the information presented in the large groups. The par- 
ticipants, group leaders, and recorders are all to be 
commended for excellent learning opportunities . 

Bob Billings ' overview of the area of job satisfaction 
and job performance helped bring into focus problems in 
this area. His comments should enable us to examine the 
handbook on vocational education measures in a more 
useful manner. The handbook provides, for the first time, 
a ready re ference to potential measures for following up 
vocational students and their employers . 

The report of the project efforts to make better 
interpretations of placement percentages provided us with 
the opportunity to see how outcome measures might be 
practically applied to a real evaluation problem in 
vocational education . 

I would be particularly remiss if I did not say a 
word about the excellent hospitality that Billy Howard 
extended to us on behalf of the Commonwealth of Kentucky 
and the city of Louisville. The tour of the Jefferson 
State Vocational School and Manpower Skill Center was a 
special " side effect " outcome . 

We are especially pleased that the National Institute 
of Education saw the need for this Conference as a part 
of our scope of work and for the inputs we have received 
from the project advisory panel. We are particularly 
pleased with the cooperation that we have received from 
our project monitor. Bob Stump. The dialogue we have had 
with Bob over the past several months on project matters 
has been informative and helpful in conducting the 
project. 

If a conference is to be a success, it needs to be 
structured so that the original objectives can be met, 
as well as flexible to accommodate unanticipated events. 
I would particularly like to commend the project staff- 
Floyd McKinney, Kenney Gray, and Marie Abram — for the 
way in which they kept us on schedule and targeted our 
efforts during these three days. 
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I would like to conclude these remarks by presenting 
my perceptions of ten important reflections on outcomes 
for vocational education that have emanated from this 
Conference. I would imagine that you could add to this 
list and would encourage you to do so. It would be my 
hope that these ideas are just the beginning of a con- 
tinuing dialogue on such an important topic. 



1. Outcome measures as see tat ed with vocational 
education are complex due to the wide variety of 
programs offered. 

2 . Outcome measures are potentially controversial 
due to the differing phi losophical viewpoints 
and values associated with vocational education . 

3. Outcome measures in vocational education should 
not be constrained by federal legis lation — 
emphasis should be given to how the outcomes might 
be used in program improvement efforts. 

4. It is difficult to reach agreement on outcome 
measures for vocational education due to the dif- 
fering levels of interest (e.g., secondary, 
postsecondary , adult, federal, state, local, 
etc. ) . 



5. The traditional outcome of vocational education 
(i.e., placement) may be interesting but not 
necessarily helpful or useful as the ultimate 
criteria due to the wide variety of influencing 
factors (e.g., economics, employment rates, etc.). 

6. Some outcome measures wil]. need to address the 
ability of the students to perform the occupa1:ional 
skills that vocational programs profess to teach. 

7. The results of evaluation need to be linked with 
planning. In the past, evaluation has been viewed 
as a one-way upward flow of information with 
\ittle relevance for program improvement. 

8. In attempting to measure the outcomes of voca- 
tion al education , we need to consider alternatives 
(e.g., systematic observation, interviewing, paper 
and pencil tests, proficiency tests, unobtrusive 
techniques, etc. ) . 



-221- 



9. It is important to question that v;hich we are about 
and not become complacent by merely accepting 
where we have been as the necessary roadway of 
the future - 

10. Vocational educators need to be able to increase 

our ability to question and to think systematically 
about the future. 

In response to George Copa' s questions as to where are 
the Prosper' s or Dewey's I say, perhaps they are within 
this room — it is up to you. 
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IV. NATIONAL CONFERENCE 



The National Conference on Outcome Measures for 
Vocational Education was held August 16-18, 1978 at 
the Gait House, Louisville, Kentucky. 



PLANNING AND CONDUCTING THE CONFERENCE 

CONFERENCE OBJECTIVES 

The objectives of the conference, in terms of outcomes 
for the participants, were as follows. At the conclusion 
of the conference, the participants will be able to 

1. identify and explain the problems and issues 
concerning the interpretation of vocational 
education outcome measures; 

2. identify alternative criteria for determining 
vocational education outcomes; 

3. develop a rationale for the selection of alternative 
criteria for determining vocational education 
outcomes ; 

4. analyze the consequences of having standardized 
vocational education outcome measures; 

5. identify the needs of significant audiences 
concerning vocational education outcome measures; 
and 

6 develop recommendations for future research and 

development activity concerning vocational educa- 
tion outcome measures. 



CONFERENCE mOMOTION 

Th'- following major promotional activities were 
conducted prior to the conference. 

1 Printing and distribution of a brochure. The 
brochure included the objectives of the confcrcnco, 
major p >sentations, an overview of the conference 
aqenda and preregistration information. The 
brochures were mailed to head vocational teacher 
educators, state directors of vocational education, 
state directors of vocational education research 
ccordir. iting units, state vocational education 
planners and cvaluators, directors of vocational 
education in L.e large city schools, vocational 
education management information personnel, 
federal government officials, and selected others. 

2 Articles in the American Vo cational Journal , 
Centergram, and Manpower Weekly . These articles 
provided conference highlights and preregistra- 
tion information. 

3 Personal letters of invitation. Over 150 personal 
letters were mailed by the project staff to 
personnel in Kentucky and the surrounding states. 
Personal letters were also mailed to selected 
individuals in all states. 

CONFERENCE AGENDA 

The conference agenda was planned around major pres- 
entations, with reactions by the members of the project 
advisory committee. Small group sessions were used to 
secure participant reactions to the presentations. A 
summary of the conference program follows. 

Wednesday, August 16, 197 8 
1:00-1:30 p.m. Conference Registration 

FIRST GENr,RAL SESSI ON 

1-30 p.m. Presiding - Floyd McKinney 

I-IC Dm Greetings from the National ^ , 

^•^^ P-'"- institute of Education Robert Stump 



-224- 



ERIC 



Welcome to Kentucky 



Billy Howard 



2:00 p.m. 

2 : 30 p.m. 
2:45 p.m. 
3:15 p.m. 



4 :00 p.m. 
4:10 p.m. 
4:20 p.m. 



Introduction of Project 
Staff and Project 
Advisory Committee 

Overview of Conference 
and Projected Con- 
ference Outcomes 

A National Problem: 
Interpreting Outcome 
Measures in Voca- 
tional Education 

Quest ions and Answers 

Break 

Value -based Premises 
Underlying the Choice 
and Interpretation of 
Outcome Measures in 
Vocational Education 

Reaction 

Reaction 

Recess 



Floyd McKinney 



Kenney Gray 



Robert Morgan 



Mary Kievit 
Ralph Bregman 
John Grasso 



6:00 p.m. 
7:00 p.m. 



9 :00 p.m. 



SECOND GENERAL SESSION 

Recept ion 

Dinner 

Master o f Ceremonies 

Eva luat inq Vocational 
Education: A Congres- 
sional Perspective 

Meeting of Small Group 
Facilitators and 
Recorders 



N. L. McCaslin 



John Jennings 
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8:30 a. 



9:15 a.m. 
9 :25 a.m. 
9:35 a.m. 
10:00 a.m. 



11:45 a.m. 
12:00 noon 



Th ursday, A ugust 17, 19 7 8 

T HIRD GENERAL SE SSION 

Presiding - Kenney Gray 

Requisites for and Con- 
sequences of Definitional 
Standardization for 
Vocation a 1 Education 
Impact Data 

Reaction 

Reaction 

Break 

Smal 1 Group Interaction 
Group A 

Leader - Arthur Lee 
Recorder - Roy Giehls 

Group B 

Leader - Donald Healas 
Recorder - Bob Andreyka 

Group C 

Leader - Ralph Bregman 
Recorder - Jim Bebermeyer 

Group D 

Leader - Robert Warmbrod 
Recorder - Martha Raymer 

Group E 

Leader - JoAnn Steiger 
Recorder ~ Jesse Clemmons 

Group F 

Leader - John Grasso 
Recorder - Bill Richardson 

Break 

Con f erence Luncheon 



Donald Drewes 
Arthur Lcc 
JoAnn Steiger 
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FOURTH GENERAL SESSION 



Presiding - Marie Abram 



1:30 p.m. 

2:00 p.m. 
2:15 p.m. 
2:30 p.m. 

2:50 p.m. 
3:00 p.m. 
3 : 30 p.m. 

3:45 p.m. 
4:00 p.m. 



Concepts and Complexities 
of Measuring Job 
Sat is faction and 
Job Per f ormance 

Reaction 

Questions and Answers 

Instruments for Measuring 
Job Satisfaction and 
Job Per f ormance 

Questions and Answers 

Break 

Analysis of Extant Data 
Bases in Vocational 
Education 

Report of Project Case 
Study Findings 

Recess 



Robert Bi 1 lings 
Donald Healas 



Kenney Gray 



N. L. McCaslin 



8:30 a.m. 



9:15 a.m. 
9:25 a.m. 
9:35 a.m. 
9:45 a.m. 
11:30 a.m. 



Friday, August 18, 1978 

FIFTH GENERAL SESSION 

Presiding - Floyd McKinney 

Criteria Against Which 
Vocat ional Education 
Should be Held 
Accountable 

Reaction 

Reaction 

Break 

Small Group Interaction 

Break 
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Grant Venn 
Alan Sheppard 
Robert Warmbrod 
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11:45 a.m. 



Con f erence Luncheon 



12:30 p.m. 



Summary Report of Small 
Group Interactions 



Georqe Copa 



12:50 p.m. 



Perceptions of Con fere nee 
Analyst 



N. L, McCaslin 



1:00 p.m. 



Participant Evaluation 
of Con f erence 



Marie Abram 



Con ference Ad j ournment 



SMALL GROUP INTERACTION 



During the Conference, the participants were assigned 
to six groups to interact and formulate reactions to the 
major presentations. Following are the reactions of each 
small group. 



Small Group Response to Paper by Morgan 

A. Some of the questions and their reactions were: 

Q: Is VEDS to be a service function or merely a 

vehicle to determine compliance with legislative 
mandate? What should it be? 

R: VEDS is a compliance mechanism. Failure to comply 
could result in curtailment of federal support. 
Feedback is needed by NCES but regulations will 
have to be followed. 



Q: Should there be a national standardization of 
vocational educati on data? 

R: This represents the ultimate, but many of the 
participants feel that such standardization is 
virtually impossible . 



Small Group A 
16 Members 
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VJill data be collected from all vocational programs 
in a state or on ly those programs supported with 
federal funds ? 

Colorado has instituted a statement of intent 
by the student, but it was indicated that 
persons weren't happy about it. One reason 
for reporting all students would be to more 
accurately determine labor market needs. 



Should there be a difference in outcome measures 
needed at the local, state and federal levels? 

Yes. lea's need the most information, state 
aaencies need less , and the fede ral government 
needs the least. 



Flow many former students and employers should be 
involved in follow-up — the population or only 
a sample? 

It was pointed out that NCES will establish sample 
designs which are optional. 

To what extent is VEDS being coordinated with other 
federal reporting requirements such as CE.r', VA? 

NOIC has been established to eliminate unnecessary 
and duplicative information . 

What does VEDS mean when they refer to a certificate 
No consensus on this item. 



/ What uses should be made of the da^ia? 

Data must be in a form that will assist LEAs 
in f)roqram improvement . 

What is the pay-off from VEDS? 

As far as vocational education is concerned, it 
will benefit from more information rather than 
suffer from it. The coercive aspects of VEDS 
should he unnecessary . 
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B. Some of the concerns of the group were: 

1. NCES may have interpreted the federal mandate 
too literally. It appeared to some participants 
that the system will be both expensive and dif- 
ficult to implement and will yield little useful 
information for program improvement . 

2. Former students are being imposed upon when they 
are asked to identify immediate supervisors or 
employers . It was pointed out , however , that in 
those states conducting evaluat ion of employer 
satisfaction this had not been a problem. 
Students when informed of the f ol low-up and its 
purposes, tend to be more cooperative and return 
rates are h igher . 

3 . The method being used to determine employer 
satisfaction may not be the most effective. 

One alternative would be to secure reactions from 
persons employing vocational completers rather 
than accumulating information on an individual 
basis . 

4. P: concern has been expressed by some states 
about reporting only those programs included 
in the state plan . One participant pointed 

out that to report others would be like including 
history or math students. Discussion revealed 
that many states have legitimate vocational 
programs which neither receive federal funds nor 
are reported in the state plan. 

5. The number of hours requirement as a criterion for 
an eligible program may eliminate reporting many 
students who are legitimate completers. 

6. Because of the many federal programs associated 
with vocational education , students may be 
reported more than once . 

Additional comments regarding the VEDS discussion were: 

1. V^Jhen NCES decides on the requirements for a data 
system, such requirements should remain constant 
for five years . 

2. If schools are really attempting to improve out- 
put data , they may stiffen admission requirements . 
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Small Group Response to the Paper by Billines 



A. Some of the questions were: 

Q: How does an evaluator know whether satisfaction 
comes from the job or training? 

R: It was noted that other elements having no relation 
to education may impinqe on job satisfaction. 
There are multiple effects present here. Far 
more confusion than knowledge exists regarding 
the outcomes of vocational education. 



Q: What are the problems attached to competency testing 
R: Perhaps this is a matter for another conference. 



Q: Should we, or can we have national standardiza- 
tion of com[)e tency testing? 

Q: Can a [profile of students and programs be developed 

that will more adequately indicate the outcomes 
of vocational education? 

Q: What about a study by the National Center that 

would identify the components of a student profile? 



D. Comments 

1. It is important that concerns regarding outcome 
measures be voiced now rather than waiting for 
them to be prescribed . 

2. We must assume that most information we get from 
local agencies is valid. 

3. Vocational education is doing a better job of 
cjatherincj cind disseminating data than any other 
area of the educational enterprise. 

4. Too much emphasis has been given placement as a 
criterion for success. 

5. nterpre tation of data is important. Many 
factors other than the attainment of vocational 
sk i 1 Is affect job pi acement . 
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6- The National Center could make a significant 
contribution by providing leadership in the 
development of a model for evaluating vocational 
programs . 



Small Group B 
14 Members 

Small Group Response to Papers by 
Morgan^ Drewes , Kievit, and Jennings 

A . Are as of Ag r e eme n t 

1. Conference 

a. Must find middle line between evaluation 

needed for local planning and evaluation for 
federal level (accountability) . 

2. Learn from Title I and other past evaluation 
systems . 

a. National systems have little utility to anyone. 

b. A national data system does not answer 
questions; Congress wants to know, "Is the 
nation better off because of vocational 
education?" 

c. Congress-needed data has limited utility at 
state and local levels . 

3. National Evaluation 

a. The die is cast; current evaluation conference 
cannot change criteria. However, a critique 
is necessary to change future evaluation 
systems . 

4. Drewes ' Paper - local level 

a. Local level units will only accept data to 
support their own value system. 

b. Decision makers must be open to evaluative 
data (agreement in group that a national 
system will not be accepted) . 

c. School disuricts planning for five years must 
include what surrounding school districts 

are doing. 
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d. Good data base needed at local level — 

process compatible to collecting national 
needs; however, a national system may not 
pick up unique conditions and program at the 
local leve 1 . 



5. State Level 



a. In some states (Ohio, Florida) , state legisla- 
tures are asking questions and providing money 
to do evaluation. 

b. Agreement that states must provide funds to 
LEAs to conduct evaluations. In this way, 
decision makers could plan and insure quality 
programs . 



6. Job Placement Criteria 



a. State legislatures and Congress do not know 
what they want, but they do want to know job 
placement information . Not interested in 
other things vocational education could do . 

b. Vocational education could be destroyed if 
job placement is only criterion used. 

c. Job satisfaction and job sat isf actoriness 
will cause problems for vocational education 
if they are the only criteria considered. 

d. Vocational education placement match to manpower 
data will shift interest to CETA. 



Areas of Disagreement 

1. Politics at local level determine all things. 
Although there was agreement in that local 
education agencies must face dropping programs, 
there was disagreement on the emphasis Drewes 
places on locals abiding with local pol i t ics . 

2. Disagreement with Drewes that only formal efforts 
are being done. Use of informal telephone calls 
within the state are common . 



3. Caut ion--pcoplc will resist most data that 
will result from evaluation efforts. 
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Areas Needing Further Consideration 

1. Definitions 

a. Groping for defiiiition "What is a vocational 
student?" 

b. Competency-based programs could vary the number 
of hours necessary for mastery. 

2. Who will use evaluative data? 

a. Legislatures and board members will ask, 
"How is vocational education impacting 
on social issues ?" 

b. State should insist that data be used at 
local program level for decision making. 

c. AVA will use data to help vocational education. 
Opponents will use same data to hurt vocational 
education . 

3. Use system that is in place. 

a. All local education agencies and states are 
providing data to federal government. 

b. Adapt system and develop a sampling formula 
(some states have large numbers, other states 
do not need to sample because "universe" 

is manageable) . 

c. Ohio and Florida must follow up all vocational 
education students who graduate or leave 
program. 

4. Purpose of vocational education. 

a. Must provide vocational education programs 
for objectives other than job placement. 

b. It is dangerous to be tied only to manpower 
formula. 

5. Strengths that should be emphasized. 

a. Vocational education improves retention in 
secondary schools . 
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Small Group C 
20 Members 

Small Group Response to Presentation by Jenning_s 

A. Congressional Expectations 

1. Planning should be done in order to make decisions 
about programs . 

2. Reduction of unemployment of youth, disadvantaged 
and handicapped . 

3. To know what happened to students other than those 
placed in jobs. 

4. To know funding mix for local programs. 

5. Placing of people in jobs related to their training 
(which IS what we say we are doing) . 

6. Additional value (of output) for added cost of 
vocational education. 

7. Reduction in duplication of programs, national 
and state . Comment : The responsibility for this 
reduction placed on vocational education should 
be shared. 

8. Long term trends to show some degree of match 
between vocational education program areas and 
labor market demand . 

9. Positive effects in terms of employment of handi- 
capped, minorities, and women in nontradi tional 
occupat ions . 

10. Employers be satisfied with the work, readiness, 
and skills that students receive. Comment: 
Methodology should include personal interviews 
with employers in addition to questionnaires . 

11. Student to be satisfied with the program. 

B. Information Needed 

la. Manpower needs 

b. Enrollment 

c. Output costs 
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2a. Special programs 

b. Work history of students 

c. CETA — ramifications 

3a. Information that could lead to reducing unemployment 
b. Follow up needs 

4a. State contribution 
b. Local contribution 

5a. Follow-up 

b. Maintain accurate data 

c. Evaluation 

d. Salaries of graduates versus salaries of 
nongraduates 

6a. More definitive statement of individual program 
standards expected 

b. Evidence that standards are reached 

c. How student achievement contributes to improvement 
of a student's status in the labor market 

7a. Evidence from coordinating councils 
b. Information from private and public sectors 

8a. Match between program graduates and labor demand 

9a. Handicapped (groups) who previously had no access 
to vocational education. 

1) enrol Iment 

2) job placement 

b. What traditional programs have accomplished 

c. Enrollment by sex, race, program 

10a. Employer survey comparing vocational education 
and nonvocational education trained people. 

1) skills 

2) attitudes 

3) motivation 

11a . Job information 

Addi^-.ional Expectations Identified in Whole Group 
Session 

1. Vocational education system to have ability to 
adjust to employer needs. 

2. Data to be used to improve local programs. 
(Group felt this was not a Congressional expecta- 
tion . ) 



3 Vocational educators to add expectations to those 
expressed by Congress, e.g., more rapid advancement 
of vocational education graduates, higher wages, 
etc. (Group felt this was not a Congressional 
expectation . ) 

4. Coordinated planning leading to adjustments in 
delivery . 

5. To be able to compare output from vocational 
education with output from other training sectors. 

6. Input should be measured (characteristics of 
students) as well as output measured in terms of 
standards . 



Small Group Response to Paper by Venn 
A. Concerns of Vocational Education 

1. Found that criteria for vocational education 
evaluation is appropriate. 

2. Vocational educators help set standards — and 
stressed that vocatioi al education is for preparing 
for jobs. Therefore, job placement has been P' lor 
criterion . 

3. Nontraditional measures — retention in higher 
education . 

a. Youth clubs in vocational education 

b. Cooperative attitude 

c. Self-concept 

d. Greater growth in co-op (in real world) 
motivation to go on. 

4. Myths in vocational education such as the following 
should be challenged: 

a. individual changes jobs five times. 

b. industry wants general or liberal arts back- 
ground and train them. 

5. Need to define how we will use evaluation at local, 
state, and federal levels. 

6. National system does not mean standardization in 
vocat ional education . 
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7. Policy on evaluation needed. 

8. Dealing with special population qroup could reduce 
placement . 

9. "What we talk about is what we are judged by." 

10 . Tasks — common denominator across the country . 
Tasks crossover to identify related occupation. 

11 . Special population a vocational education will 
affect placement. 

12. Criteria for disadvantaged bi lingual . 

13. Special needs need to be considered on data 
collecting and evaluation. 

B. Issues in Vocational Education 

1. Identify outcomes for other state. 

2. If we are judged on placement, there should be 
funds to do it. 

a. Placement service in each school 

3. Placement forms should obtain more data on 
technical skills. 

4. Purpose of evaluation should be to help local 
schools. 

5. National policy clear on special populations. 
Small Group Resporise to Billings 

A Questions of Concern to Vocational Education 

1. What are the key determinants of job satisfaction 
and job per forinance ? 

a. Salaries 

b. Job advancement opportunities 

c. Security 

d. Fringe benefits 

e. Skill levels 
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However, all of these items could change day- 
to-day dependinq on many variables sucn as family 
responsibilities, i.e., single individual being 
at homo or married individual with family re- 
spons ibi 1 it les . 

2. What compltjxit ies in measuring job satisfaction 
and job performance should vocational educators 
give particnlcir attention. 

a. Age of student 

b. Self sufficient 

c. Married or single 

d. Health 

e. Personal aoods 

f. Attitudes 

Note : We are not taking exception , but are 
basically accepting Dr, Billings' premises as 
out 1 i ned in his presentation . 

3. What rationale can be provided for using employee 
job satis fact ion as a vocational education 
outcome ? 

Need to look at assumptions under this criterion, 

a. Training affects job satisfaction 

b. Student selection of vocational education is 
a free choice 

c. Student selection of vocational education is 
one ne/she continues to think is a good one. 

4. That impact of training on job satisfaction is 
constant and cons is tent . 

5. [Measurement is valid and rel iable • 



Small Group D 
'T8 M embers 

Small Group Response to Paper by K ievi t 

A, Introduct j.on 

In addition to the audiences mentioned by Dr. 
Kievit, the group identified two additional audiences: 
the media because of their impact on public relations, 
and the employer, who is the key to the system. The 
cjroup suggested the following student expectations 
for a vocational education program. 
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1. Freedom to I'.ave a choice 

a. of procjram 

b. of career 

2. Develop saleable skills 

3. Achievement and success (important in student 
value system) 

4. Recognition 

5. Acceptance by peers 

6 . May enrol 1 because course is popular (auto 
mechanics ) 

7. Mciy be taking a vocational education course because 
it is the only alternative to a total academic 
program. (Example given: Kansas, 187 schools 
which offer agriculture as the only available 
vocational course ) 

It was iicated that post secondary students are 
more mobile than secondary students and therefore 
may have more freedom to choose vocational courses. 

(' ■ comes Students Need to Know 

The group then discussed competencies students 
need to have. Competencies suggested were: 





Awareness of labor market 




2. 


Job opportunities in the 


fie Id desired 


3. 


PrediL:tions for possible 


future emp loyment 


4. 


Competencies domLinded by 


the job 


5 . 


Supply and demand data 






There was a lack of total 


agreement on the answer 



to the question: "to what extent should students 
know in advance tho competencies necessary." Also, 
the group disagreed on "good" students get jobs. 
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Other points and/or quest ions brought out by the 
group were; 



1. Is it possible tnat when students are learning 
skills, that they also learn about job opportunities, 
etc. ? 

2. The job information should be made available to 
the student prior to enrollment in the vocational 
program. 

3. Students are not locked into a vocational program. 
They may change fields late in the program* 

4. How can the data system give credit to the voca- 
tional education program when a student finds he/she 
is not suited for that occupation and makes a 

wise decision to change? Example given: 

nursinc; field. This should not count as a failure. 

There was consensus that former students should 
be asked if they achieved what they expected to and 
if they are satisfied. 

Some members of the group felt that competencies 
should not be measured at the end of the program 
unless they were measured at the beginning of the 
program . 

Some of the small cjroup participants felt that if 
vocational education is to consider the values of 
students we need to know what students expect then, 
we need to find out if s tudents are satisfied . 
Those who disagreed noted that adolescent student 
expectations may not always be correct according to 
the values held by society, that adult input may 
affect student values, and that student values change 
as the student becomes more mature. One example 
given of possible change in job satisfaction was if 
a youncj man's qirl friend moved to another town, he 
could become dissatisfied with the job because of 
this factor. 
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Employability vs. Employment 

The group expressed concern on how to rank 
"employabi lity " as an important outcome . Quite a 
discussion ensued on "employability" versus 
"employment." The group agreed that employability 
and employment are not the same and should not be 
used interchangeably. They indicated that employ- 
ability is a function of education, is a major factor 
in program design, includes a whole package of 
attitudes , skills and competencies , and helps to 
determine if the product of the vocational program 
has become a saleable commodity. 

The question then arose regarding how employers 
look at this. "Students will have to become 
assets, not liabilities." To what extent do 
employers have to be satisfied? What is success? 
Is it 100 percent? Can students satisfy employers 
70 percent, 40 percent? How can we define the 
"level of success?" Could we aggregate data, see 
where we are today, then determine percent of in- 
crease? If today is 35 percent, could we move up to 
50 percent? How could one account for the total 
number of students involved if this system were used? 
Employers are concerned with the "effectiveness" 
of the person hired. 

For one group of students at a given time, a 
certain number were employed. How does one determine 
if this is good, bad or indifferent? The cost 
analyst and the vocational teacher would look at the 
data differently. Should we be satisfied if employers 
are satisfied with one out of three students employed? 
Employers want students who have a good attitude and 
are willing to work. The employers answers may differ; 
however, one may find that when talking with the 
personnel director attitudes are stressed and that 
when talking with the assembly line foreperson 
skills are stressed. 

Some significant discussion centered around the 
federal law, its regulations, and how we should respond 
to it. A point was made that we should discuss 
vocational education philosophy and make it paramount 
rather than always referring to federal regulations 
and law. 
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D. Packaging Information for Various Audiences 



Considerable discussion centered on packaging 
information for each audience — students, parents, 
counselors, members of Congress, employers, etc. 

I t appears the same data can be collected and 
packaged di f ferent ly . It was noted that vocatior.al 
education needs to work on improving public relations. 
Properly used data could be a great asset in improved 
public relations. 

One group member reported that a congressman had 
recently requested anecdotal type information, such 
as "Is the student on co-op program? How much money 
was earned? How did this affect the economy of the 
community?" It was thought that' if Congress had good 
anecdotal information, case studies, or testimonia Is 
about what goes on in his/her district, and if some 
of this were published, it might have more impact than 
all the numerical, statistical-type data. 

E. Packaging Information for Federal Legislators 

The final statement was--legislators respond to 
organized pressure that can affect votes — numerical 
data could back this up. 

Small Group E 
13 Members 

Sma ll Group Response to the Paper by Kievit 
A. Areas of Agreement 

1. Expectation (values) of students when they elect 
vocational programs may vary from that of ad- 
ministrator or parent. 

2. "Placement" criteria will not apply to postsecondary 
students who want to progress rather than place. 

3. Definition of "student" is central problem 
in evaluation. 

4 . Paradox is trying to de f ine vocational education 
so as to fit federal funding rather than in the 
broad context of occupational training. 
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5. Mistake to use "youth unemployment" argument to 
gain support for vocational education because it 
will haunt us during evaluation. 

6. Be pragmatic in reporting data to Congress. 

7. Evaluative conclusions must be related to other 
factors, e.g., labor market, economy. 

8. Any competency measures must be based on realistic 
objectives . 

Areas of Disagreement 

1. Whether to use placement and/or/nor competency 
as measure of effectiveness. 

2. Vocational education should (should not) accept 
goals other than job training, e.g., self- 
concept , attitude , etc . 

3. Vocational education does (does net) have an 
image problem. 

Areas of Further Consideration 

1. Problem of short-term gain (CETA) as opposed to 
long-term gain (vocational education) . Which 
philosophy do we adopt and defend? 

2. What are the expectations of the various publics 
we cater to? 

3. What is " relatedness?'* 

4. Is charge to serve special populations counter- 
productive ? 

5. Are we claiming too much or too little for 
vocational education in our public relations 
efforts? 

6. Where does public relations fit into presentation 
of evaluative results to various publics? 
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Small Group Response to the Pap e r by Venn 

A. Areas of Aqreement 

1. Academic students should be compared to vocational 
students . Floweve r , one should con side r-- 

a. me thodoloqica] [problems , 

b. interveninu variables, 

c. variance in vo^n:': ional problems , 

d. probl'^^m of in :g "treatment" provided 
by vocatiC'Ur . -.rrxira^/ and 

e. problem o-^ ^. :.on into each type of program. 

2. Purpose~~goals native criteria form a definite 
continuum in v«..c iL onal education. 

3. We really have c^-^';ided what vocational educa- 
tion is; therefcr^^, w can we evaluate or compare 
it to other areaL" ? 

4. In settin:j critoric:, need to consider the 
indiv idua.l di f f ej '}nc3r- of each student . 

B. Areas of Disagreement 

No areas of disagreement were recorded. 

C. Areas of Further Consideration 

1. Should the goals for vocational education emphasize 
economic development or student interests, needs. 
The answer varies by state and locality. 

2. One criterion of vocational education might be 
improvement in basic skills. 

3. Is vocational education considered a separate 
procjram or simply a component of the overall 
oducat ional system? 

Areas for Add it i onal Research and Development in 
Interpre ting QiTtc ome Measures for Vocat ional Educat ion 

A. Clarify implications of new legislation for evaluation. 

B. Develop instrument to allow noneducators to evaluate 
vocat ional education. 



C. Clarify vocational education, purposes, qoals, and 
objectives. 

D. What is a comprehensive program of vocational 
education ? How does it fit into overall education 
system? 

E. How do characteristics of entering students affect 
success ? 

F. Examine concepts that have served to influence 
legislative decision making. 

G. Clarify definitions. 

H. Develop national measuring scale. 

I. Communications from vocational education to outside 
audiences (power groups) . 

J. Aspiration vs. expectations of different audiences, 

K. Evaluative criteria and methods that concentrate 
on quality . 

L. Ways to assess outcomes from a variety of perspectives. 

M. Are we a "method of teaching" or a "discipline" in 
vocational education ? 

N. Ways to convince publ ic that vocational education is 
a mainline activity. 

0. Are educational managers trained to administer voca- 
tional education programs? 

P. Effective ways (methods) of teaching vocational 
education . 

Small Group F 
16 Members 

Small Group Response to Paper by Drewes 

A. Concerns of Group 

1. How to deal with outcome of student, concerns for 
employment and unemployment factors. 
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2. Present data too qeneral, specific data too costly, 

3. Need for standardized definitions. 

4. Need for participation of labor. 

5. Placement not crucial, need to develop students; 
job is ireans not end. 

6. Need for standardization system of collecting 
placement data across states. 

7 . Data needs at f edera 1 and local levels di f ferent . 

8. v;ill \^DS aenerate qood data? 

9 . Vocational education has multiple and conflicting 
aoals . 

10. Collection data for compliance reduces its 
cf fectiveness . 

Small Group Response to Papers by Venn and Billings 

A. Why Are Wc on the Defensive? 

1. Conqress has challenged us. 

2. Arc wc overreacting? 

3. Can vocational education impact all social ills 
(unemployment, minority equity)? 

B. Report to the President 

1. Information from states 

2. Conflict: brief report or an exhaustive report 

3. Reports from states, some poor 

4 . Local data also very shaky 

5. Still compliance 

C. What Is Vocational Education 

1. Mult idiscipli ncs a curse (one opinion) 

2. Multilovo], I.e., secondary, postsecondary , adult 
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3. Employment and training vs. human perspcc:ive 

D. Definitional Problems 

1. Vocational education vs. occupational education 
(occupational, vocational, technical, career) 

2. Titles 

E. Vocational Education Needs a Taxonomy 

F. Where Does Problem (Vocational Education Lack of 
Leadership) Lie? 

1. SDE/teacher education/local schools 

G. Local Level Perspective 

1 . Confusion 

2. Decisions can be made at local level 

H. USOE Data 

1. 1976 legislation based on 71-72 data 

2. 1976 summary data not too useful in 1979-80-81 
planning 

I. Hurdles in Utilizing Data 

1. Are data helpful? (They probably are not, locally.) 
J. Evaluative Research (crucial) 

1. Dete nining net effect of vocational training. 

2 . Comparison data . 

3. Mil this type of data have any effect locally? 

4. A qrowing up process. 

5. Effect of vocational education on productivity. 

6. Conception issues raised by psychologists. 
K. Political Considerations 

1. Must become politically astute and stop all the 
philosophy about definitions. 
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L. Systematic Evaluation Needed 

1. Eva luation process must have a system, especially 
for states who are just beginning. 

M. Invasion of Privacy 

1. VEDS data, will it invade privacy? 

N. Outcome Data 

1. Concc rn over lack of input or feedback on ongoing 
research to develop evaluation models. 

2. V^cre are we to look for direction. 

3. Concern that all research comes from one source. 
Regional research is needed. 



OPTIONAL TOUR 

Through the cooperation of Billy Howard, Kentucky 
Assistant Superintendent for Vocational Education; 
Bill Evans, Director of Vocational Education, Kentucky 
Region Six; and James Woodrow, Principal Jefferson State 
Vocational-Technical School and Manpower Skill Center, 
two optional tours of the Jefferson State School were 
provided for the conference participants. 

Located within the inner-city of Louisville, the 
Jefferson School was completed in August 1977 at a cost 
of nearly ten million dollars. The sciiool has 157,000 
square feet of floor space and is designed for 3000 
students. The school is offering twenty-six vocational- 
technical nroqrams . 
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CONFERENCE EVALUATION 

The conference staff designed an instrument to evaluate 
the variety of activities at the conference. The instrument 
i s shown in Figure 2 . The evaluation instrument was part 
of a notebook of ma^cerials the participants received when 
they registered. Participants were asked to complete the 
evaluation instrument and return it to the conference 
staff. Early leavers were also encouraged to ro:T»plete the 
instrument. A total of seventy evaluation instruments 
were returned. 

The evaluation of the conference will be reported in 
the followincj five sections: 

1 . con fere nee p lann ing and implementation 

2. presentations 

3. small ciroup sessions 

4. the obicctivcs of the conference 

5. cfoneral comments and recommendations 
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Fi gure 2 

ConferencL..* Evaluation Instrument 



NATIONAL CONFERENCE ON OUTCOME MEASURES FOR 
VOCATIONAL EDUCATION 
August 16-18, 1978 

CONFERENCE EVALUATION 

P^.ovide. ijoiiA ccuidid reipoK6c-i to the (^otfoiving quQ6tion6 
about t/ic conf^cAi^ncM, T/ic inf^onmation be used to 

jyiovida {iUcdback to conf^cAQnca plmncAyS and pla^ente^^ 
to oTipYc'i'c (^iitiiM! QndeavoK6. 

P^ACKGROUND INFORMATION 

Check your major responsibility. 

State director of vocational educ ation 
Local director of vocational cducatiui^ 
State Advisory Council 

Staff 

Member 

Voci^tional planning and evaluation coordinator 
Placement direc tor/counse lor 
Vocational program specialist 
Vocational teacher educator 
RCU di rector/specialis t 

Other (specify) 
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Ficjure 2 , Con t inued 



CONFERENCE PLANNING AND IMPLEMENTATION 



Please rate the overall quality of 
the individual areas of the conference 
as compared to typical professional 
development conferences you have 
attended. 



Quality 



Pre -con fere nee in format ion 



c 

0) 



Q) 

u 
w 



o 
o 



u 

> 



2. Meeting facilities 

3. Room accommodations 



4 . Me a 1 s 



a. Reception . . 

b . Dinner Banc^ue t 

c. Luncheons . . 



5. Conference organization and 
operation 

6. Usefulness of conference 
materials 

7. General sessions 

8. Small qroup sessions . . . 

9. Conference as a whole . . 



10. Conference location (Louisville) . 



Coirunen t s : 
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PRESENTATIONS 



Please rate the effectiveness of the presenter and thv 
4uality of the content presented by placing a chock (/) 
in the most appropriate box in each column. 



E f f ect ivenoss 
of Presenter 


Quality of 
Content 


1 

Excellent i 


Good 


CD 

tr 
fd 
j-i 

> 
< 


1 Fair 


Poor 


Pre sen t. at i on 


Excellent 


Good j 


1 Averaae 


u 


Poor < 


f 

1 










1 . A National Problem : Intor- 
p re ting Outcome Measures in 
i Vc - :at ional Educat ior 




















2 . l\: ? - >ecti vism in Choos inq 
an c. I n t r p re t i n g Ou t c ome 
Measures in Vocat i onal 
Education 












1 











B.Evaluatinq Vocational Educa- 
tion : A Congressional 
Perspective 













4. Outcome Standardization for 
Compliance or Direction : 
The Critical Distinction 























5 . Job Satisfaction and Perfor - 
maiiCe Measures . The State- 
of- the--Art 






















6 . Instrumentrj for Measuring 
Job Satisfaction and Job 
Performance 












; 










7. Analysis of Extant Data Bases 
in Vocat ional Education 






















8. Ref)ort of Project Case Study 
Find i, . s 














1 








9 . Criteria Agai nst Which Voca- 
tional Education Should Be 
Held Accountable 













Comments : 
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SMA LL G ROUP SK SSIQNS 

Please rate the effectiveness of the interaction that took 
place in each of the small uroup sessions in terms of the 
objectives of thu sessi' . . 



01.^ i c c t i vo s of Se s . i on 


Excellent m ,o 


ual 

nte 

0 

o 


-it 
jra 

(U 

(T3 
U 

> 
< 


y c 

cti 

u 


)f 

on 
u 

0 
Oh 


To analyze and fjrouaro reactions to the 
[jfL'sunt at ions by Morcjan , Kievit, Jennings 
and Drewes. 












To cinalyze and [.^reuare reactions to the 
[:)resen tat i ons by Billinqs, Venn, Gray, 
McCaslin and McKinney. 












To develop siicKiestions for assisting 
various audiences Lo interpret outcome 
measures in vocational education. 













Comments : 



OBJ p:cTivEr 



How t.' f f oct i vt' ly did the conference 
Did thio conference assist you to 



meet its objectives? 



Extreme ly ^ot 
We 1 1 Adequate ly at a. 



.Identify and exf^lain the prob- 
b..'ms and issiir-s concerning the 
interpretation of vocational 

education outcome measures? 

, I de n t i f y a i • r n a t i vo criteria 
for de te rm in m-i vocat i on a 1 
education outcomes? 

. Deve lo[j a r * t . (;nale for the 
se lec t ion o " a i to mat i ve 
criteria for di^ t--- rm i n i na 
v( ^ca t ion a 1 od uca 1 1 on out come s ? 
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4. Analyze the quenccs of 

having standardized vocci- 
tional education outcome 
measures? 

5. Identify the needs of 

significant audiences con- 
cerning vocati onal educa- 
tion outcome measures ? 

6 . Deve lop recoimnendat ions for 
future research and deve lop - 
men t activity concern inu 
vocational education outcome 
measures? 



Fxtromt^ ly Not 



4 3 2 1 



4 3 2 1 



COMMENTS AND RECOMMENDATIONS 



1. What were the stronger features of the conference? 



2. What wore tlie weaker features of the conference? 



3, What suaqestions would you make for improving a 
conf e re nee of this type ? 
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CONFERENCE PLANNING AND IMPLEMENTATION 



Ton dimensions of the conference planning and 
impleinen ta t i on uore rated by the participants using 
a five point scale (excellont/cjood/average/f air/poor) . 
Room accommodations received the highest ratings with 100 
percent of the responses in the excellent and good categories. 
Preconference information received the lowest rating. 
Summaries of the ratings for each of the ten dimensions are 
shown in Table 3. 

The followincj comments were received concerning 
conference planning and implementation: 

The diversity of backgrounds of participants made it 
difficult to secure agreement on issues in the small 
(jroup sessions . 

Usefulness of conference minimal as my state already 
has a system. 

Excellent plannincj for conference. 
Excellent facilities for conference. 

Well ornan ized , very aood time framework , interesting 
topics , and certainly a timely and informative con- 
ference. 

Appreciated lodcjinq within walking access to the city. 

Conference dwelt too heavily upon philosophy and did not 
develop guidelines for measurement of vocational 
education program outcomes. 

Small group sessions too highly structured — no room for 
personal input. 



PRESENTATIONS 

The effectiveness of the presenters is revealed in 
Table 4. As can be noted, very few presenters received 
ratings in the fair and poor categories. 

The ni:. ' presentations were rated in terms of the quality 
of the contc^.t of the material presented using a five- 
point scale (excellent/good/average/fair/poor). In all 
inGtances far more than i alf of the ratings of all presenta- 
tions were either excellent: or good. 
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Table _3 

Percent of Participants Rating Quality 
of Conference Planning and Implcmeiitation 



Conference Planning Quality 

and Implementation Excollont Good Average Fair Poor 



Pre con fere nee 
information 



19 



44 



15 



12 iO 



2. Meeting 

facilities 



6C 



25 



Room 

accommodations 



84 



16 



Meals 



a. Reception 55 

b. Dinner Banquet 55 

c. Luncheons 62 



29 
38 
35 



5 3 
2 0 
0 0 



Con ference 
organization 
and operation 



44 



7. 
8. 



Usefulness 
of conference 
materials 42 

General sessions 2i/ 

Small group 

sessions 26 



47 

55 

43 



11 
16 

19 



0 0 
0 0 



Con ference 
as c'l whole 



37 



57 



10. Conference 
location 
(Louisvi lie) 



54 



33 



ERIC 
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Table 4 



Percent of Participants Ratincr Effectiveness 
of Presenters and Ouality of Content 



Effectiveness 
of Presenter 



Ouality of 
Content 



1 4 



2 6 



54 



2 3^6 



60 



24 



Il2 



19 



45 122 



29 



40 ^0 



50131 



I 



20 



20 



46 



22 



4 712 5 



47i2 3 



5 0 39 8 



Presentation 



l.A National Problem: Inter- 
pre t i no Outcome P-e asures in 
Vocational Education 



2.Porspectivism in Choosina 
and Inte rpre t inq Outcome 

a s u re s in Vocational 
Educat ion 



29 



3.Evaluatina Vocational Educa- 
tion: A Congressional 
Perspective 



0! 



4. Outcome Standardization for 
Compliance or Direction : 
The Critical Distinction 



7 0' 



13 



5. Job Satisfaction and Perfor- 
mance Measures : The State- 
of- tlie- Art 



6 . Instruments for Measuring 
Job Satisfaction and Job 
Pe r f ormance 



0 

t ■ 



7.Analy£-is of Extant Data Bases 
in Vocational Education 



RGport Project Case Study 
Findings 



9. Criteria Aaainst i«'hich Voca- 
tional Education Should Be 
Hold Accountable 



16 



48 



2 4 



13 



17 



1 4 



19 



55 



53 



44 



36 



53 



50 



54 



51 



35 



22 



21 



13 



16 



32 



22 



29 



24 



10 



The f 11 owing comments were rece ived concerning the 
effectiveness of the presenters and the quality of the 
content of the presentations. 

It would have been helpful to have more information 
on the ideas concerning how to mea^vrc outcomes, 
evaluation models being developed b/ NIE, Contract 
Research Corporation, etc. 

Some of '-he papers required study before presentat ■ >n. 

Any one of the presentations could have been better 
explored and understood as a conference in themselves. 
I marvel that so much was accomplished in so little 
time and am thankful for the printed materials provided. 

Presenters should have the opportunity to respond to 
reactors. 

Good reactors, well chosen small group leaders and 
recorder.'. . 



,MALL GROUP SESSIONS 

Three objectives of the small g-.'oup sessions were 
rated by the participants using a five point scale 
(excellent/ good/average/fair/poor) . Instructions on the 
.^valuation instrument asked the respondents to rate the 
ob jec"*-.i ves in terms of the effectiveness of the interaction 
that took nlace in the small group sessions. 

The second objective (to analyze and prepare reactic^-s 
to the presentations by Billings, Venn, Gray, McCarlin, 
ind McKinriey) received the' highest ratings with 29 percent 
being excellent and 45 percent being good and no responses 
f dl 1 ing with in the poor category . 

Tl J third objective (to develop suggestions for assx^ -'.ng 
various audiences to interpret outcome measures in vocat. j;ial 
education) received the second highest ranking with 6 
percent being excellent and 48 percent being good. 

The first objective (to analyze and prepare reactions 
to the presentations of Morgan, Kievit, Jennings, and 
Drewes) ranked third with ratings of 9 percent excellent, 
38 percent qood, and 36 percent jveraye. Table 5 sum- 
marizes tho qual i ty of th-: interactions of the smal ] groups . 
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Table 5 



Percent of Participants Rating the Effectiveness 
of the Interactior. for the Small Group Sessions 



Object ives 
of Session 



Ouality of Interaction 



Excellent Good Average Fair Poor 



To analyze and 
prepare react ions 
to the presen tc.t ions 
by Moraan , Kievit, 
Jenninqs a'ld Drewes. 



38 



36 



11 



16 



To analyze and 
P re pare react ion s 
to the presentations 
by Billinqs, Venn, 
Gray, McCaslin, and 
McK inney . 



29 



45 



13 



13 



To develop suggestions 
for assistinq various 
audiences to interpret 
outcome measures in 
vocational cducacion. 



48 



23 



l"/ 
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The following comnieiits were received concerninq thj 
quality of the interaction in the smalJ aroups. 

Time too 1 imi ted . 

Not enouqh structure provided by 1 eaders . 

Too mucli structure provided by the leaders. 

Need for better trained leaders. 

Suggest speakers circulate amona small aroups. 

Need to spend more time aett iPicj acqua i nted with other 
group memloers. 

More time should have been spent on devc*lopinci standard 
measurinrj sticks and def init i ons . 

The backgrounds of the smal 1 group participants were so 
diverse that it was difficult for participants to work 
ef feet i ve ly together . 

THE OBJECTIVES OF THE CONFERENCE 

The participants were asked to rate how effectively the 
conference met its objectives using a five point Likert- 
type scale. Table 6 summarizes the participants' ratings 
of the conference objectives. The conference objectives 
are placed in rank order according to the ratings by 
part i'*^ i pants . 



COMME: TS AND RECOMMENDATIONS 

Three questions were asked in this section of the 
evaluation form. Each question was open-ended and will 
be reported on separately. 

One of t'v questions was, "What were the stronger 
features of the conference?" The responses centered cn 
three features: organization, presenters, and printed 
materials. Positive comments on the oi gan J. zat ion were most 
frequent. One statement appeared to sum ujo the feelings 
of all of the respondents who listed organization as a 
strong feature. This statement said, "This has been one of 
the best oraanized meetings T have ^^ver attended. The 
agenda , as distributed, wa' held to, meetings started 
on time, '^nd sessions were paced we] 1 and unhurried." 
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Table t. 



Porc^nt or Partic' 
the C'b iect i vc* s of i. 



nts Rat iri'i 
. Con f erence 



Conference 
Object ives 



Extremely 
Well 



Adequately 



Not 
At All 



Identify and cx[jlain 
the problems ar:d is- 
sues concerninq the 
interpretation of 
vocat ional educat ion 
outcome measures . 

Ident i f y alternat i vo 
criteria for de- 
termini ncj vocat ional 
education outcomes . 



49 18 



10 



36 34 19 



Develop a rationale 
for the selection of 
alternative cr i ter ia 
for determining voca- 
tional education 
outcomes . 



28 46 19 



Analyze the conse- 
c]uences of havina 
standardized vc -ational 
oduca t ion outcome 
measUi.'^es . 



10 



28 43 12 



Identify the needs 
of sicinificant 
nud ioncos conce rn i nq 
vocation a 1 oducat i )n 
outcome measures. 



30 39 17 



Dove lop re commend at ions 
for future research and 
de ve lopmen t act ivi ty 
i'oncQ rn i nq vocat ion a 1 
educat ion outcome 
measures. 



14 



22 25 24 
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Presenters were the next most frcqucMitly mentioned stroncj 
feature of the conference, and printed materials was the 
third strongest feature of the conference. Small nrouf) 
interactions, conference facilities, and the choice of 
topics for the conference received favorable coiuments from 
the participants. 

Another question that was asked was, "What were Lhf^ 
weaker features of the conference?" Small qrouf^s were 
cri t ici zed most often . The criticism of ""he small a roups 
centered mainly on the shortness of time and the lack of 
structure given by leaders. The next two most frequently 
criticized features were the [jiresenters and M^e lack of 
solutions presented at the conference. Presenters were 
criticized for reading their material and attenjjtinq to 
present too m.uch material in the time allotted. 

The following suggestions were provided in response 
to the question, "What suggestions would you make for 
improving a conference of this type?" 

It is a difficult area to deal with~~felt the 
conference staff did as well as could be done. 

Incl Jde more varieties of people , e.g. , legislators , 
Department of Labor, handicapped, ethnic minorities, 
etc . 

Distribute papers before formal presentations or if 
possible, send to participants a week before conference. 

Need for more interaction among participants. 

More fociis on "what to do" instead of "should we do." 

Fewer general session presenters and reactors with more 
time being given to the ones selected. 



Have presenters circulate among small groups. 



SUMMARY 



The conference was evaluated L-; seventy participants. 
Nou all participants responded zo each item. Therefore, 
data reporued are uiven .".n f orcr-ntaoes to allow comparisons 
among items. 

The overall results were as follows: 

1. The conference planning and implementation were 
generally rated highly. Room accommodations ranked 
first with 100 percent of responses falling in the 
excellent and good categories. Preconf erence 
materials ranked lowest with 63 percent falling in 
the excellent and good categories, 27 percent in 
the average category, and 10 percent in the poor 
category - 

2. The presenters v/ere rated in terms of (a) their 
effectiveness and (b) ^he quality of the content 
of their presentation. All speakers received 

over 60 percent of their ratings in both categories 
(effectiveness and quality of content) in the 
excellent and good categories. Therefore, it was 
concluded that all speakers were highly effective. 

3. The small group sessions were rated lower than the 
prL'senters. The lowest rated objective (objective 
one) had just under 50 percent in the excellent and 
good categories. Objective two had 74 percent in 
th? excellent and good categories and objective one 
had 54 po cent in the excellent and good categories. 
Comments indicated that limited time and low group 
structure interfered with qroup interaction . 

4. The objectives of the conference were ranked using 
the five point Likert-type scale. The objectives 
ranked as follows: 

• Identify and ex])lain the problem.s ar. ' issues 
concern ing the interpretation of vocational 
education outcome measures . 

• [den t i f y al ternative cr i teria for determining 
vocational education outcome s . 

• Analy ze the consequences of having standardized 
vocat Lona 1 educat ion outcome measures . 
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• Develop a rationale for the selection of 
alternative criteria for detorminina vocational 
education outcomes . 

• Identify the needs of significant audiences 
concerning vocat ional e ducat ion outcome measures . 

• Develop recommendations for future res'.^arch and 
development activity concerning vocational 
education outcome measures. 

Comments and recommendations given by participants 
indicated that oraanization , presenters, and 
written materials were the stronqest features of the 
conference; the- functioning of the small groups, the 
presenters, and lack cf solutions to problems 
discussed were the weakest features of the 
conference; aud a more diverse population, distribu- 
tion of papers bofDro presentations, increased 
inter act i on amona part icipant s , more focus on *'what 
to do , " fewer presenters going into more depth , and 
having presenters mailable to small groups for 
questions were th<: recommendations given most often. 
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V. VOCATIONAL EDUCATION MEASURES: INSTRUMENTS TO 
SURVEY FORMER STUDENTS AND THEIR EMPLOYERS 

NEED FOR THE PU3. ICATION 

The National Institute of Education -nd the National 
Center for Research in Vocational Education identified the 
need for informaLion on methods and instruments available 
to collect data on altern-:^c-Lve outcome measures in voca- 
tional education. The n.-^ed for the handbook resulted from 
(1) the Vocational Educ^ ion "Amendments of 1976 which 
recjuired additional information concerning the outcomes of 
vocational education, (2) the concern for accountability, 
(3) the concern for the individual student presently 
surfacmc: within the /\inerican culture, (4) dissatisfaction 
with plac ^.ent data as the sole criterion measure of the 
success of vocational education programs^ and (5) the lack 
of availability of a similar compendium of instruments. 
In response to this need, a handbook tif -T-d V ocational 
E ducation Measures : I ns truments to Su_rv'';v " ^ormer Students 
and Their Employers was developed. It io designed to 
provide to vocational educators an assortment of instruments 
currently available for use. 



PROCEDURES FOLLOWED IN* CONCI- T'TUALT Z I NG THE PUBLICATION 



liTuiiedi ate ly following the decision to produce a handbook 
of instruments hel[)ful to vocational educators on alterna- 
tive outcome measures in /ocational education, the decision 
was mado to collect instruments in four major categories. 
The four categories are as follows: 

1. .Jol) Sat I ^5 fact ion , Instruments which have items 

tTia't" ^ rfrV- '"pur f)or ted to measure employees ' percepti.ons 
of the i r sa t i s f act i on wi th components of their job 
((}.q., fjay, r)romoti' s, supervision, co-workers, and 
work) and in which trie sum of th^-- scores across 
corn: 'onent.s y ields an overal 1 job atisf action 
measure. (Instruments which on^. ; have questions to 
measure over;>ll job satisfaction were not included.) 
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. Jul) ! r t (; rinanv'r . i ; i ^ !" i 1. v.'!iic;i luivc 

that av'^ ini r !.){ )r t" ( ' ( i t. f ) ri'MSur^r t: !u' !" I orrMiici^ 
♦.^mr.O oyot^.s i.n tht-" \Mri<"'U;" i iru'iu-. : (\ns/''cf.)miH-.;p.ont 1' 
v :Ta p I u y ni<.j p. t ( t.^ . . ] . , k : i < 1 < • : ^ i ^ , ski 1 h ' , a t: t, i t u ( k > 3 , 
.irui wcjrk a t t • Mu::tiu ■■ - ,^ j :*.rc'..MV«Hl ^/ thr onin 1 c-'V (. r . 
( I ri s * f i.inu.'ii :> whiaa i>Miy aavf^ auost. ions In m<\"isur'^ 
'■j.'*aj\ili !..'ini ; .1 ( )yf ' i ^ ^ ra"'. a'niaiK't ■ w-aa- not. i ruM aa-.'a . 

. i^va'caa L c. :• 'I'rM.inLiaj by I''ornua*' Stucicnt. 

last :a.iniant:s wli.iL^ii iiav*.' il^.Miis that, ar-' r>ura(^r ' i ■> 1 
to ni'^asaL''.' the pcM-cH^pt- i raa; of fornica" s t ' ul< ai t. 
?"i M : a ri 1 L I'M ] tlw.: a ( it:"^aj.iaay of cc^rf^oncMi t s of t ho i 
rrain iaia pro'iraiu! (a . a . , Laci i i i.cs arul ccj u i.; ^iitLai t , 
com I o t a a c .i. a s a c c pa i ra. A , a na I | 1. < ' clmiio n t s r v i c a a ) . 
( I :iatruniorU:s wh Lch on] y h.ivo qucvst ions (■:>n th*? 
oV'j ra 1 I. aaoau.'.iay cjf t lu* trainin.a prauiram wnc not 

■; . la . r, .;PMori (.a" 'i'r'iin.ina by I^inployor. T n s truncal t a 
vvaia:; hava- it^'ins tiiat a ra"^ piirportod to iiioaaura 
\.]io "trocti aanoss of compononts of {)rcparatirm 
ana ti'ain.i.iip f ca' om|.j loynie n t (c.u., knowlaxlcK- nnd 
skill, aroas) as pajrcoi.vod by an om|.)loyer of porsoms 
•a a. ) i r-ccn 1 1 y ac^iri].' Ic^ tccl a job t r^a .in .1 ncf p laocj ram . 
1 i as t ]-anicai t s v;hich only have c|aestions to rjoca-^ur-^ 
ca.aa.'a 1 1 adt^cfuacy ol proparat.ion and tra i n i. np 
pa.'oarar.is wt.ro not included. ) 

VJi.th t'aos<.' :aaar cat(jc|orics in mind, the |.)rojcct staff 
solicited iais t rumL:n ts currently in use in these four 
catcqcjrios frc^m (1) vrKantional educ^ition research coordi- 
natiruj unit .iirtaa.a)rs iat a I .1 f.ifty states, (2) directors 
oil vocational "du(;atLon in 200 large cities of the nation, 
(3) industrial ::'sy cdaoloq i s t s in selected major universities 
and industrial ccjmpanios, and (4) professional, orqanizat ions 
in industry includinq SIMCON and Mayflower. The staff also 
searched comj^ute r i zcul data bases includinc] (1) central ERIC 
and the clear i nqhouses serving career education, community 
junior colleges, and tests and measures; (2) the library 
of the National Center for Research in Vocational b'ducation; 
(3) dissortat ion abstracts; (4) rjsy chol ocj i.cal abstracts; and 
(5) sociolr)qi ca L abstracts for copies of instruments. Instru- 
ment source books such as I3uros Menta l M ea su rement Handbook 
v/ere also r(_>vi.ewed for pcassi.ble instruments. 

Th(/ cr i. terL<i ased to describe the types of instruments 
beincj requested and selected for the handbook were (a) instru- 
ments whicn m'..'r't lao d^a-K:r i ;)t i ')n of one or more of thc^ fnnr 
ins t r union t. s 1 i s ted abcvv/(.> , aivi ( k) ) i ns t r amen ts whi ch cou ] ■ i 
be read and. und^aastood by a former hia'a school stucUaU'. 
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?.\A': :'■ iy yr, ]C'rt sl::i:'r t:^' iit'I:' 

• * ^ i i ; ; ^ r unit.; n t; . 'Vhe :ibstraci:s 

* ■:' : n.ii.^ i i. :^ h^.- r for additions, 

, • /■. : ill^'ii . i; ':;r'")Va 1 v/as re- : ; i-'^ s lit.- 

i: : ■' ::: : :;. i v/vjTO included in th^- 

7: :■: .:. •* ;" n!^;^ - n r s vn..'rp aijs t r act t?d and 



W' obtainon by tno 

: t o iiu.'^; a t ]. ua s t (")n(:;' ( > : 
a ■ ■ "] a rat.al y (.lap li ca t i 
'::\ i S:^ 1 on t o :jr i n t the 
d:'.' d<.'\a;' I(.a :c r , ' v^uL' Iishor . 



.1 1 - 



':d and as a r"esu] t msi: r 

i :: a. an i. r^aaTl format 
:a.j ii anaoook . A -"Oi_a' : r 
i* ■ S'S r i be oacii of '■ :\c 
a in arn va n .i. r; Fi a a :."o ^ . 
i-aa ao t "ir"id i ncludea '-.]\c 



'.[.'■ i': >.-.': I ' : aa-os'' i.> T a uiiinia r i z i nq per tin' : 

r:r. i\na- n : • i a. .: a: ; ::;a av. 'i'i^^^ adstrart:a also sarv*,-' 
' . vP' ; r" i L 1 L'S i : -a.-ip^.-a ..a- t;;u iiandbcuak , "How to 

PS'.- liaa.aa ." \\\ aa/pa- !- riw--.', a seri.os of. c]UOStions 

; : - • p • a' a : • • : : ' ' in an /a. t^ "inp tc"^ ].» - ad the 

; aa;-a V 1 a.-:':* a ' a : ■ a ; . n a" • : - -a .a ^ d' ■ a i s i on -:nak i nc| p rrK:oss 

'doiv. '.v: a,; : d d' d ; a "p : ■to; a\.\ ^ a.; pj c t i v'o s and ioad to 
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A"\iLAn: lATY : 

}''or I'^i! r ^ iiT^ r I i; To r^inat. ion: 
!■: R 1 C : 

L)::sc^'' I I'Ticx ' .:• i ::s''':^i;Mr:::T : 

Cont'?nt : 

N umbo r o f I " : 

1)0 of ] t. '?nis : 
in t i^oou I a t l : 

Kcs: -oru.lo:! f. : 

r,DAi:i I, S':'i-:A':" : 

Whon Adn^i M i s t.* : ro- 1 : 
How Acini i n L s to ■ ro' i : 
'i' i ' ! • ' c : (;;rnriu: > n ( i 's r ; : 

Scor i no : 

^^'::::t i;Ai7^: 

Ro I i.ib 1- 1 i to/ : 
Va 1 i.d i '.y : 



i:. :'":u r i a 1 s sent to 

. ' . I ■■jy 'I "^c i ^ C\ in oth(.?r 
urc'.:-:-;. St -coi: ;-]ary or sub t i. t: 1 arc 
t. :v<jr.\ the rr:)in title by a colon, 

•n: t Mr rior-^' mstrunients had the 
I'w- \ : b. , >ii r" forcnt.i-ition was made 
nh' i : n. : C i n oaron theses ) zhe names 
' h - ■ ao'.^r^ c* 1 us C'T authors and the 
' ' ■ ^ 1 o: ^t:>(i . 



nui'i^- o:' th'-' indiwi duals or 
■ :■; ; c -n s l i :> 1 e for t h o de ve 1 op - 
I h- L ns t rune n t . 

: n wii 1 oh the i.nst rumen t was 
\\ ] [ e n t h IS info r m a t i c n was 
bbi bo ttiO entry read, "not 
.: . " Copyrights arc also noted 

'Ct i on . 



■ fubl mailin'-: af.ic:ress of an ir.divid- 
i or" a'.:i'ncy r-ospons i 1) 1 e for the 
/'J 1 r.^r-men t , r.jub lication , aiid d.istribu- 

o f t:i";r- L n s t r ume n t or the comp le t e 
> 1 I our iibn i.ca.l -."^i tation to the 
' rn.i t. wr:-:- . 

o o of . i n s t ume n t to Individ- 
I s . ? ur.' s 1 1 n cj it. I n mos t i !' s t arice s , 
u; f h' ■ i !is t runven t is not ava j. lab le on 
'omi-V' V c I a .1 b:\s 1 s , the user should 
^oct to ;-)ay normal reproduction costs. 
1 man y i ns t an ces , aqe ncies we re 

u t an t t o I ) 1 ace s c i f i c costs on 
* i r mat'.M'i als due to the unknown 

o i :o I :n<':tors tiiat may affect the i ]' 

.1 i : iiumbej]' f^.o: document i:; 

'h hn-.'- b.M.M r.la^-'Mj i i ; t-HTC. 




;)F i :;s'rHL:>:i::N':' : 

•:. • ' ; ■i.j I .i i what 




;j f I Litis : Tiic ir^L.!; i^uv^i:^':' i t > sns (Ui 

: ii s t r un* ' M L . A i i ^> t. i n < ; c.i f 1 1, : i ui,' i ■ r ' ' t 
i ttuns ill t'arli sLi!-' ra-M.or w i 1 ] a 1 :w • A' ' 
A : 'Xi t'.' A vAi< ' rc' , i : : ■ ;" ' --A ^' i ^'^ ' - 

^A.-::-- ]*.••:■.:: 'Aic ^:ir^:o? of ita^'i::; (< -A-, ^ ri i^' - !"< i 1 S' • , 

rr;; A t : p ^ c cho L ci? , A i k( ' rA- - ! yp< ^ r^^S[ cm , 
i rho ..omc )i.is cA) (.") i C' ■ , ^ ' c , ) \ nc ] wd^^d . ii 



^t.;^^ FN rS'iAA-/"] '.A 



tluA:. 'h'-' nc^asurc^ was d'a.siunr-d Im ,iya.-;i'ss 



-fj r s on ( '.^ . cj . , s t uci' mi t: , fo rrrv,.' t' 
af , t.oach^jr, ranp 1 oy^.^o , < 'nii .A < j v r A 
a unp A A- ( ' : ; t: !"i ' ' i n s I r umo n . 



AAnnir: i a' a !aa.l 



TAa limc (date) suqaca^ta-ci t,o adni i ias t a ' r 
' :\<j iiu^trumont re td to enrollment 
\ . a . , upc;n o nrcj^l 1 inq i n voc:a t i can a 1 
a r oe r am ) , c j r a d ua t i on (one y c a r a f 1 1: r 
aradnation) , c^r employmerU". ((.^.a. , 
• 'Hit A cv./r M.i in a job for four months) . 
\ [- not 1 cAmi t ^. f i '-'d th- ^ ♦ - n tay' rericA; , 
" not- ! lid i ca teai . " 



ilOW 



T:aj na.uvier in wh i eh an instrurusit was 
d'''.">ian'/d to bo adm i i s tf • red (e.a., by 

, 1 ; ] ^ [: r ' 1 r > : ; \\oVi < ' \ n t . f ^ T V , f a o'a f f a cl 
int."rvi''W, or \n c:roup sa^ttincj) . If" 
:. r a. : d< a A 1 f : ' ;d ! h. ' ^ a A ry • ads , 'Aio t 
:rd;a'at(^d." 



I-', -:3 t-haz shoula be 
L 1 r:.: the ineasuro . 
t- try I'eads , 



.r/ ; l c:i : J i */ i Jua 1 s provico 

r I s tc 1 t c: ni s i n c 1 u do d on. the 
r'.-^ i '.: . v: . , K.aper-penci 1 , individual 
r s , consumable booklets , 

If not identified the entry 
, " :u ' y i ny] i ca ted . " 

::■ : r-/ f'. rs to both ( I ) the ?iathe- 

- 1 : rooedure for calculat incj 

1 d: ua 1 score s ( e . . , s imp le summa - 

va:* I obo:ed resf.-onses , etc. ) and 
iir ■ : -i-iys icr 1 me t!nod of scor in.g tiie 
{('..,;. , ruiiid s cor inq , op t ica 1 
K'tc. ) . Information on the 
nt' L lity and cost of scorinq 
c e -s , v; h e n a \^ a i 1 a :j 1 e , is a 1 s o 
.led i.n this section . VJhen not 
e e-itry reads , "not 



■ \ ::.:':> \ \ [' ■ ■ 1 n : : 'rr;:i r. i f ri related to the consistency 

■! d( y -.:ndai) i lity) witli v;hich a set of 
d.,ita measure v;rtatovcr tliey do measure. 
1 riOt identified the entry reads, 
":"::.it 1 i cated . " 

'.',:!: d L ' y : Th.*' -iccui'acy vm th which a set of data 

I'^easure wiKi t they are said to measure . 

If TiOt i.c:t.'nt. 1. f led the entry reads, 
" n(;t i nd i ca ted . " 

'MMKN'rS : Any Icited information reqardinq 

t' ' r i a 1 s ^ar back ci round i n f orma t i on 
assoc: i a ted v/i th the f ol low- up me a sure s , 
wh icii wa,s not included in previous 
^ct I ons , i s Lncornorated here . 



Loach mc: in 



: :: L : 1 1.' A r t , " ^'^-^ s v/ r l t to n by I' ob^: ■ r t: . 
,^t,n^■n:. of Psycholoc;y at The Oh i o Stnto 
1 i ricTs ;ois clone extoni^ L vc^ rc* so arch ano 
o:' oatisraction and job per furniancc^ . 

, :;.it 1 s fact i on is do!" mod an; 

) r : I'l t .1 1 i o r i t o v;a r d o r onio t i o n a J. 
react, ion to tho job (or various components of 
LP.o ';ob ' ]--nnnltino f roni tiio ap{;raisal or evalua- 
tio^n o: on^^'^ job or '^r\, co-njo r ionco (bav;l.cr b) 7 3 , 

h';0>^■ T-^Tt ) , 

^v\i\ - c:o:^ni_tion ^>I:lohas r zo 5S tv;o in torroia tod points: (1) job 

^cti'^n a toolinc; or an emotion and (2) tho fooli.ncj 
of- ^sa^is^-ict i(;ni is dorivor.i from a (more or loss) careful 
(-valuation of one's jot). Three models of job satisfaction 
are described: Lawler's comp^onent model, person environ- 
nent fit mod-:-], and th^^ met e>n-)ec t a t ion n model. 

[..avbl-e- nodcoL sriov;n in Idigure ^ suciqests ti"iat the 
;^niouril o ' lob natisfaction results from an evaluation of 
satisfaction with one's job is a function 
the' di ncro'r.'ancy between what the person feels should 
and wait is seen as r o-: i s 1 1 ncj ) . Pay is an 

A:. empLoyee wor.ld be dissatisfied with pay if the 
' l^voe felt siie/h': should be compensated more than she/lie 
ovv^i' ' t h- ' "shoubi be" and "is" W(-rr^ m balance, the 
emoloyo!:^.voulo i.. ■ satisfied. li the pay received was cn-oater 
than th^^ rnaolova': b-alievod she/he should (jet, a feelinn Oi 
; ^ 'tov^'e a. a;;: ■ -..s.d-. L^ad' in'Oiults . Th is co-:amp]''' 



tiio icxo 
of 

e X 

f'^nip 



"siiould ijc' 



ictiori 

A a > b dissa::sf ■ctio!" 
/i a ( ij aailt, ino'.'i'ui!;;. 
/ : d 13 com foil. 



Perceive a an^:Oi:nt. i) 
that "is" 
received 



anizations. Belnont, CA: ';.'adsworth 



2 



1 r^; 



i c p u r s o n / o : 1. r - o n ni o : 1 
ares t-hai: a poi'son won hi 

i J- r 



o • r . u a r : 01 : 1 :: 
■ i:; L 1 cios , ;ir 

:h.'ro or lo:^^^ 



oocomo 1 ri' -o r t a r". 



rircu.uat:iO' la .1 aan 



:uns I c:^' r a t: xoas , a ac , 



X o 1 o r t ■ a 1 ri 



^ t i oaa I odaca t: ac^uid L>o orad - c tad 



I a c raja s- 



^atiara 



^idK' ti.j rd modal rr' ser^ t" O'd^ called mot C' x; )OC t.a t oa a . 
rocas^;S ar: er;u)loyee a xacK: t a t i oa s . In this model, chi' 
erosp-^ica i -.'o (jnaaloyc'a iais a sat: r)f expect.it 1 ona about what, 
the job shoald off-'/r. 11 idiose expectation?.- are aot rrn^i., 
rhea diaa e: i :■ tac'- : or; aad, - ' vea t aa 1 1 y , turnover resulta. 
'I'h i 3 modL- 1. la not ^a uaeful a a i-awl'"^r'a br^oaasr^- the <^auaea 



Tl^e ''cd. :oWi. na tear' j.>oas i. b b.^ 
voCfitioaa! i v,a.i(a': t oai ai"^"; r] >r'..ai 



^^M:a r)t -joo a 
cdiant^'r tv;o : 



1 n 



Voea 1 10a a 1 r,'d ',aa"i t. ion n iajh t lead to b-e t te r j 'jba 
fru" araduates, hc' t^^r job outcajmca^;, arid hi.ala-r 
aa t i. s fact :ajn . This ^irqument reste^i on an 
inpaor tan t ass amp t ivon- - tha t "be t te r jobs " 'AO^re 
a\'a .1 1 ab 1 and Wf:'re niorc^ J ikely tr^ be oh> ta .i n':.''d: 
r o 1 1 f ;w i n ( J q o o ( .1 V o c a t i o n a 1 e d u c a r. 1. o n , a s c < j rn pa i r ( ' c I 
t o r ; o o a t r a i n i n c i o r.' n o a e a t a J 1 . I f r e 1 a t e c: 
1 ' ) ij v: ^ ■ r •■ ca v a .i. 1 a b 1 ti or if t h*",' c h a n c e s o f 
c ; t f , ; I'a J related j cj }_> s w a s n o t i m r r o vc:* d ia y a^ ; m ~ 
t. i nii "i 1 e duca ta on , t hra 1 th i s h ypo t\\o s i s v/fau I d 
ru.; h)L^ r;or n (_ai t . An iinr^l i ca t i on v;a s t ha 
i ra.: r' :-a S'..: d loij sa t .1 s tact i on niiciit dc^fjoraj just 
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■vui 'l oducrit ion 

: ■ o V a 3;ji ra c lor; ; 

v.'.t :^ thri t be t: i.o r 
lc:^-.;c 1 t:^At i.s!'.-ict ion- 



5^:iou .i.a 



: ;::t . ■ small. , m:k^ t.o 
J ' : s f . i ^:t: ioM . A ] tiio liv)h 

] ,:it M";''! J roct: i. affc^cL some 
■ ' ; : < : ; ^ ^'cMnonh , t he re wg r c 
r vao les wh 1 ch v/oel-:! 
' te ; r*r^. i r ; a n t s oi joo 
: . \' ; 1 t lori , vocm t: ion a I 
; :;■ ' r ^-a a.; sat 15:; fact len 

.■a^.., na*.-' wo ,:{fccts on 
: ■ tS': 1 s fact i or'^ v;i. tli 

•1;'..-;, ail*: to t.aaj -incrtj^sed 
. . ; a::: : i aaaa; a a a;x?ve . To 
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G. The leisure ethic. Work is nocessary, 
but can never be fulfilling; because 
f ul f i llmen t comes from le isure activi- 
ties; wor k should r oq u i r e less t ime 
t rian i t doe s . 



In chapter two, it is noted that job performance is 
a more complex va^" ^^^le t!^ ^^ job satisfaction. Job perform- 
ance is not see- ^t , but rather as a whole 
clust'^r of cUr ..apter notes that when 
peiiormancG s usually assessed aV .le 
number of d.. -ften include such t 
the f oJ lowin ..antity, skills, techi 
kn.owledcje, abj^xu^ o v. ,rk v;ithout supervision, aLuu^ ance, 
punctual ity , cooperat ion , coirununication skills, problem 
solving skills, teamwork, in torp^ersonal skills, safety, 
good attitudes, and initiative, cirr.ong others. 

There is no (jno current list of job performance 
dimensions for liII jobs. 'I'he ideal approach is to do a 
complete job analysis for a specific job and derive an 
instrument for that job. At the other extreme, a small 
set of very general dimensions could be applied to any and 
all jobs. A manageable middle ground is to select an 
instrument specifically dev^eloped for the job being 
assessed or a similar job. Another approach is to select 
an instrument which contained dimensions believed to be 
meaningful, as judged by someone familiar with the job. 

Job per f orinaiice is generally thought of as determined 
by three basic variables: motivation or effort, skill 
level, and role conception. Effort (how hard the employee 
work^-) and skill level (whether or not the employee has th^ 
skills to do the job) are self-explanatory. Role con- 
ception refers to the employee's idea about what should be 
done on the job or how the role should be played . I f an 
employee has an incorrec t notion of how the j ob should be 
done or what activities constitute high performrnce, then 
actual performance will be low, even though the employee 
might have the abilities and be working hard. 

These three determinants of job performance are thought 
to combine multiplicatively in affecting performance. That 
is, if one variable is very low, then performance will be 
low. Total lack of job skills means low performance, no 
matter how much effort is put into the work. If the 
motivation is lacking, then performance will be low ^ no 



matter what the skill level or role concept. In order to 
analyze the possible effects of vocational education on 
job performance, we must determine which of these three 
factors mi;;ht be altered by vocational training. Billings 
notes two possible effects of vocational education on job 
per f ormance , 

1. There are many causes of job performance. 

This seemingly obvious point must not be forgotten , 
for it implies that the effects of even very good vocational 
education wc ' '^l b'" 'iff^-ilt to discern because of "error 
variance." c. us factors could be controlled 

(e.g., ir characteristics, organiza- 

tional rtrwa. ing aptitudes, job experience, 

other training; , l. cts of vocational education 

could be easily and unambiguously determined. In any real 
life study, these uncontrolled factors may create so much 
"noise" that no effects can be seen. 

2. There are several ways in which vocational 
education may affect job performance. 

On a brighter note, this analysis suggests several 
possible mechanisms through which vocational education 
could affect job performance: increased motivation, better 
job skills, and knowing what the role entails. 

In summary, chapter two contains a word of advice on 
the selection of research designs. The "ideal" true experi- 
ment is discussed and reference is made to the impossibility 
of carrying out a true experiment. Contrasting the job 
satisfaction of a group of vocational education students 
with the job satisfaction of their nonvocat ional education 
co-workers is presented as the best possible alternative 
design. A second suggested design is to contrast former 
vocational education students with former nonvocat ional 
education students using measures of individual character- 
istics such as age, years on job, etc. A third alternative 
is presented: the contrast of vocational education students 
against standardized measures of job classification or 
demographic categories. A fourth suggestion is to combine 
several of these alternative strategies to offset the 
strengths and weaknesses of each . 
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Cha[)t(:r ti.v>'^' , "liov; tn Thi^^ ilanclbook , " is an 

attoin[;t to Mol[^ Liie practitioner cle to rmmo his/her 
obioctivcs (Quickly ami select an apf^ropr iate instrument. 
Chapter three contains a four-c^-Ll matrix of the four 
categories of instruments c'Jiitainecl in the handbook. The 
matrix illustrates that the f(;ur categories match the 
respondent (either former studen. t or employer) and what was 
being evaluat^?d ([^resent job or vocational education train- 
ing). Figure u illustrates tiio four cell matrix and 
indicates color codes used to differentiate instruments 
(color cod' instri:ments a!"e in parentheses in each cell, 

L'ha[iter ' > [^resents th'.: abstract format already 

descr ■ report (see I'igures 3 and 4). 

, ciiap'ter Liircje describes the use of single 
and doub. risks that may appear in the upper right 

liand corner of tiic abs t r.icts . A single asterisk signals 
tile usL^r of t.ne handbook th.u an instrument contains items 
that measure aspects of boti'i vocational training and job. 
Double asterisks indicate that the instruments are part of 
ci series of instruments. The name of the companion instru- 
ments appear in brackets below the double asterisks, 

Tiie bulk of chapter three is concerned with walking 
the user throu^'h a series of questions that are designed 
to help clarify the user's objectives and eliminate all 
but a few instruments for final selection* The series of 
eleven cjuestions are built around the pertinent i.nformation 
summcirLzed m tht^ abstract of each instrument, 

T hx e 1 e v en < i u ^..^ s t i. o i s a i* ( : : 

1. Dcj I wish, to (lathor information from former 
vocational education students or from employers? 

2. D(j I wish to cjatiier information concerning 
fac''^.': of tfie preser^t job or concerning the 
prev i ()u:j vocat ional education training program? 

3. Do 1 want to Locate instruments that survey 
botli em[->l(:yters and former students using instru- 
merU:s that are designed to be used together? 

4. ^/.e ne r^a 1 I y , what is thtO luuture of the information 

I w ant t o CO 1 1 e c: t ? 

5. miany items does the instrument contain, what 
type o f 1 t e m s a r used, what i n s t r u ct i.o a s a re 

: i -1 ) [ :\r- r - ' s pM:r n t , and how 1 ong will it 
Kike t:ie rt^spondeiit to comple^te tiie instrument? 

-20 ^- 



When should the instrumoiit be aJ.nini stored? 

How do T want to administer tho instrument 
(e.c;., via mail survey, telc;.)honj interview, 
in a qroup sottincj, etc.)? 

What pa[jerwork wi 1 1 be invo 1 ved for both t!u' 
respondent and the oval ua tor? How much wi 1 ] 
e va 1 ua t i o n cost? 

How va 1 id i s t b.e meas ure ? 

flow reliable is the measures? 

What other i '* >rniat i on. is availcibie on eacbi 

instrumc'- '.*^t: t-...r understand i.ts 
de ve lo^ 



! ' 111 t"i 



■■ '( ip. t . 1 1 III m! 1 :\ t h" I landHouk 



l\j HP.' ; r 




-r ^(1I7Tl<iIK■^• 
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CHAPTKR VulR 



Ch.-ipter four is the actual proscn tat i, on of the 
instruneii ts aiO;itj witli tiu-ir r uispt.^ctive abstracts. The 
instruments :Mj[n?ar i I'our colors: blue, yellow, green, 
anci orange. 'I'hese colors ccjrro^:['ond to the four categories 
as [previous.'" irulicate'd m I-'Kun't.: 6. 

Inst r uinen t s m ig h t ap[)ea r uiirjci r more than one category . 
This ha[;pens when an instrument measures aspects of both 
thic job and vocational education training. Under this 
coridition the instrument a[;^.jears cw^ ^e and has a single 
asterisk in the up[)er rnjl^.t haiul corner. 

Instruments iiug'ht also a[)poar with double asterisks 
v;i)icii, as [^r- » ^ - ^ ^ • :s 1 y described, indicate that they form 
a serir. .truments. \^'hc names of companion instruments 

apj'f' ' * s be Lo'- ' -^^e risk p. 

. . /i;j';nti:d i n s t r umc,'n t s hax-e been protected by 

overlay ;;rintirvj of "DO NOT COPY" and a referer^ce to request 
the v.aiblisher for copies and i nf ormatioii . 

'Vht: a:;|.j'jndix contains tiirt:?e related ins trmnen ts all 
of A'iiich are [.^art of a sei'ic^s of instruments. They are 
i.n:cluded to [-u;L.^sent t'nc full intent of the instrument 
p.ACk:b.;c^ of wi^. i ch they were part. 

Five ir^dices art? included in the handbook to help th,r^ 
usei* readily locate in.'U: rumc?n ts . The indices list the 
](> indi.vLchjal i ns t rumc\n t.s i)y : (1) author of instrument, 
\2) ti.tl'-* of j ns rur[\en t , (3) category of instrument, 
(4) ins tr 'anion ts forming' a series (instruments with a double 
asterisk), and (j) i ns t rumer^i ts on which validity and/or 
re I 1 a b 1 1 1 1 y h v e b c^t.^ : ^. a s t a b 1. i s h e d . 
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METHODS AND PROCEDURES FOR 
CONDUCTING THE CASE STUDY 



Tills section will, doscriljv the methods and procedures 
for conductinq the case study ^.)\ differences in job place- 
iTiont rates reported by states. S|.)ec: i f i cally , it will 
furtht.^r describe the issue ar^jas discussed in the body of 
the report, ..uul the ciata collcctior-. anc . ' ^ ' ' ■ ■ 



1 A_K_[-AS 
ASSUMP'i.' IONS 





con s 1 cit. ■ r" i 


no ta.: ai 


f f.c roaices assoc late :! with place 


i: a t 


es rt.'port 


t"- i :r/ s t:a 


tes , the f ol lowing assumptions 


rnac 


ie : 








There a . ' 


a l.i rae 


number of factors contributing 




to t h^: < : ! 


f f ^ aaaice 


1 M a cement rates . 


( ; 


All of t 


■ f a^; t (j r s 


contributing to the difference 




m placr-. 


■at rate's 


cannot be addressed in this 




study . 








'Vhc rtj nuis 


t luj a 1 o 


^jica] and clear set of criteria 




by which 


to focus 


on the important issue areas 




CO nee rn i n 


a P 1. acom( ■ 


■ t data . 



Tiie assumptions provideci the foundation for the effort 
UL-'Scribed herein. for example, tiie first assumption is 
simply a so 1 f *v i cion t truth tiiat the number of factors having 
some impact on placement rates is endless. It quickly 
becomes obvious that there must be overall logic and cri- 
teria in. limitiiia thi'C numljor of issue areas upon which the 
study could focus. 

Tiie overa 1 1 1 oa i c for f ocusincj the case study was 
derived from an attLMTipt to view candidate variance phenomena 
associated w.itli p.Kicenei^.t data ivith.in the context of overall 
vocational education evaluation. Given that this context 
IS very larae anri tia^- r*.' la t lonsi^ i|j among placement data 



phenomena are not well, unde rs LockI , i. t. was necessary to 
identify and use criteria In screerunq ur selectinq tlie 
phenomena to be addressed by t; he study. 'I'hese [dKmcMncMia 
were ex[.)rossed in issue areas. 'rhe Jocjj c and c'^'iteri<) 
usc?d in dc^ te rini nine] issu«.' arLV'is are shown, in I'i.qure 7. 

CiU TERI.A ! ^ 1 d-C'JM NM ; 1 SSIM;; ARKAS 

The major e valiKit i ^'^-^ '^It'inent.s i.n vc:cationai eehic^ition 
cvaluatien: were viewcHi uvl.i neat Iruj (elesi qnincj) , collectinu 

and provLe^nq data for mak i nq decisions. r,,Mr 
of centrality, [>e rsis tenoi^ , ""^^ - 

]. . CtM^. t ra .1 i t y 

'I'hc' ! ssue sc lectt.'d had In appear to be a t the 
cc^re ( ) t!aL' problem. I-'or exam[Ue, it was thouqht. 
that t[ie elt ' r i n i t ion of ti.^rms was central to (he 
deterinina t ion of tplacement rates . 

2 . Pc' r s I s te nee 

Tiiu LssuL^s had to a[^[^.-'ar to be those which were 
iiavint.} cumsit imfKiC L a^-: we 1 1 as endur ir^cj i.m[.jact . 
[•'or L'xam[jLe, lack oi' coMiiputer su[.)[K)rt is a problem 
t i^at may p.ot iu.' lonq-ranqt._^ i n nature . 

3 . I-'eas i. b i 1 i t. y 

i)nu. ccjul'd not i unore the [jractical matter of whether 
(jr not an area would lend itself to i n ves ti cja t ion . 
If the [;ro[)OSe<J issue area was one of great 
pc;litical significance or of a jurisdictional nature 
i t may not have been feasible i'nv the National 
Cy^nti.'.v to provide^ adoc]uate attention to this issue 
area. In addition, tl^e limit^'d resources available 
for Liris phasL.^ of the [project ^ rohibited the selec- 
t i ( n\ o f a 1 1 [nxss ib le issue areas . 

4 . R e r V a s i v c n e s s 

The issue areas selectL^d had to be areas tfiat sh(^w 
prr^mise of bc.d.ri(| wid(^spread and nc3 1 limited to a 
sin.i 1 I Murnbe-r o f s ta t . 
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The four criteria listed above were u.-'Xl to selcxM, 
issue areas that were addr^-^sed by the stu-. , . 



rssui- ARHAS SI- li-:cti-:d 

The criteria discusst^i above [;rovided the rationale 
for selecting issue areas to be addressed by the ^'cudy and 
for rejttcting other issue areas. The issue areas and the 
rationale t.;r selecting an.d rejectinq candidate issue area.s 
f o 1. low. 

Back(| rcHind 

The background or context within which voeitional 
education ls conducted may vary considerably across s t.a tes 
and may affect th.e delivery and evaluation of vocational 
oducation programs. The priority and approach to the 
delivery of vocational educaticT^ may oe addressed at the 
secondary, pos tsecondary , and adult levels through the 
existing comprehensive educational institutions, or through 
specialized area vocational education sci^ools or centers at 
the secondary level and technical institutes or other 
specialized schools at th.e pos tsecondary level. State 
vocational education leadership is organizationally located 
in the state department of education under the state board 
of ecucation or under a ::,er)arate board of education. The 
role of the state agency in enforcing the state and federal 
rules and regulations and/or providing creative leadership 
receives varying levels of em[)hasis across states. Evalua- 
tion may be viewed and treated as a requirement for 
compliance [purposes or as a means to improve loca.-^ and state 
programs and [urogram services. Financing of vocataonal 
education includes local, state, and federal support ^n 
varying percentages depending on the tax and financial 
initiatives of each local school district and ^>ach state. 

The unemployment rate v.iries across local areas and may 
influence [)lacement of vocational students. Perha[)S place- 
ment IS less difficult in areas with low unemployment ^and 
more difficult in areas of high unemployment. Some of the 
central, pervasive, and persi stent aspects of background 
i7hTc]ra"ccounted for its selection as an issue area were the 
delivery system for vocational education, the state organi- 
zational structure, the level and sources of financial 
support, the percentage of students enrolled in vocational 
education, the attituces toward evaluation at the local and 
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wit.il \"oc..; t: 1 v.^: la.I ec:iu.*a- 
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1 :k: c}ca 1 levels to do fine 



at,.^::La:a 1 ly cniorqc over t Lmc , 
• 'd M I 1 on a 1 pr ac t i ce s iri 
I 1 , :iOc i al , and economic 
; vvM\ 1 dove lo[jmen t by ieq i s- 
1 y in a k e r s a i do d b y admin- 

. 'I'iri.s process occurs 
:,x 1 sc!u;)ols and state 
):\ v;it.hin varyinc] time frames, 
vs participate in the de ve 1 - 
i . 1 r l:' to be used in the 

data . The def initi ons 
mus t be commun i cated to 
Mcal levels v;ho are to use 
t^o; t.ie^'-e loped concepts and 
:ie (iefLnition within and 
caia,^ f'ul effort to accomplish, 
i cu] l.y wiien com[3letincj federal 
se:^, or for all information 
t use of definitions 
tiie accuracy and consistency 
I Ti the collection of voca- 

rhe dovelo[)ment , 
t i o n s a s so c i a t c» d w i t h 
, i t J, ' V t.^ re s e 1 e c t e d as an 
. v c ) s I n r o 1 1 o w - u [) data 
a ri}a to placement data , 
V 1 1 me , t h e^ pervas i venes s 
.;at ion , and the hicjh 
t ' use o f de f in i t ions could 



The process oL' collectiriq voc.m t. icMia .1 e J.uccU. l on plao'-' 
menL. information involves many actors and cond i l. i ons . In 
some situ.ation.s information on p 1 .ic-ainon t ami r(> 1 a t edne.s .•; 
lb supplied by students and in otli-T situations it may b<- 
supplied by t.eacl; rs, atimi.n is t ra t or s , counS(>lors, ;ut.et- 
visors or sccrt-ta ies. i: t would appear that all o! tdvs- 
individuals miqht approach such, a task from very diflercnt. 
perspectives and orien tati o!U. rosultinq ir. C( jus i de rab i e 
differences in reijorted data. Sonu.; aciencies i;rep':-"'rc 
detailed quidclmes for data rcportinu, others hnvo wry 
limited guidelines. The ',echni.cal assistance, includuui 
monitorinq, available to aqency personnel api^ears to vary 
ureatly in its appropriateness, dep'th and timeliness. it 
aopeared that the resources available to conduct mforma- 
ticn collection activities would contribute substantially 
lo the ade(piacy and accuracy of placement data. Ihe process 
of ccjlloctinc! infoi-raat ion was pervasive to tlie field oi 
vocational education. It was feasible lo study the effects 
the process of collectinq information had on different 
-.lacemen^ rates, as it could be ol.served both .hrouijh 
dialogue witii tiie actors and rc'Vicws of related documents. 
The proc.-;s will continue as a persistent and central 
concern to the total eva 1 ua t ion ' e f f or t s of vocational 
educa t ion . 



! ' 1 a ceme n t I' un c tion 

The functi.on of lob Placement has been emerqinq as a 
responsib M i t V cjf schr.ols'and continues to be debated as to 
whether it is' m fact a responsibility that is to be pov- 
formed totally by the school, supported by the school out 
performed by other agencies, or not to be sui^ported by tf.- 
school '^^^-^ position taken by a local or state vocational 
odacQtion aqency on tlie ulacem.ent function may be ccnti-_al 
to the differences in placement rates. The strategics and 
resources used by an agency for placement puri^oses will 
indicate the relative importance given to school placemc:nt 
s-rvices The placement concern is pervasive as it is a 
national concern expressed in federal legislation for more 
than a decade. it is perceived as q concern that will 
oersist over the foreseeable future. The iUacement issue 
a?iT""i^emed feasible to be addressed as 3t could be studied 
by means of a written description of placement services 
and by interviews with key anrl related personnel. 
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ri,V;:uiremen t of the Education 
a.-niopi t data have been legis- 
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■^■t iLions; of program st:rengths 

I iyy state and local admin- 
s uc.h as the e f f ect iveness 

I-rl.anninq process , and the 
other "targeted" interpreta- 



{>iac^-men t services . ThesL' ane 

tion. of placement data were believed to be the basis' for 
fun-aing decisions^, :;.-rsonne] r.nvard structures and other 
iianagement decisions. 'V:u. :::u.,:iiAc utili::ation of the data 
IS speculated to be an influence on the diita rtported, 
irtLcuLarly where in.diviau.iJ judcfments are required or are 

' 1 i deni 1 1 f y ir. ; the various uses 
: rr>:,,; hut e: la i ienqing as there 
■ It: im})licati ons to be addressed, 
'/ere believed to pervade the 
commun i t y . 
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J SS U L AKb A: ) 



^^■hl-idi'Id) FOR INVESTIGATION 



i n.e issue are vn: w- )t. 
L ) soc i oeconom i (.: i ac to r s 
ta tt.' a - jency n ra. i n i lai t ion, 
: \ s >j two 1 s s u e < i r u ; a r t ■ 
he differences m rdacemi 
u t i n V O' s t i g a 1 1 n . ; s o c i. o e l * ; 
-i c t: c J r s a s i s s ue a r t:^ a .s v: a > 



i ■ Li 

"inc 



lor investigation were: 
i n-depth examination of 



t '//ouLd take to make 



a 



(2) 

'I'he i-.a can oe little doubt that 
impcu-tant and may contribute to 
-It rates. The chief reason for 
Knc^mic and acjency organization 
: tine apdu ic ipated length of time 



t was not 
lewpoint to 
robab 1 1; t.hat 
tate CO state 
rea ter stud*/ 



feasib l( 



a-o[ 



f t/om , i 

inc lude th.esL- 
agency (jv^uin i 



t art t I t; wou 1 d. 
than th'.^ :;r 



investigation of the areas. 
pr(j lect resource aval lability 
factors. rt was also very 
'/at ions are so different from 



ri 1. 



icult to explain without 



^'ct could perform. 



1 sue i r- 



d[.-MMARy 

''->^'d 1 n tile stud'.' were 



ssu^' ;\r* 'a 



Ma^'rCM- 



EKLC 



Issue Arc.'a I I : I>.'t'irii I io!\s m !' K'.^y Tcr'ni: 



rssuo Aroci 1 i: I : Proct^ss of Co 1 I < 'ct i PiM iMaconuMit Data 
I'ssuc^ Api.v\ i,V: !M accMiuMi t I''uiic' t: i cm 
Issue Art-a V: U t. i. 1 i ?: a t: i.c^n o\' Data 

More specific '[uostiions ;ideiroas. 'i.l ^~n\' L'acii issu<.' ai"t.vt 
ILow. 

^)\jKS'Vio:^s AiJDKi;sSi':i) vrv issui-: arka 

The s[)ecirLc aiiest.iaMis acKi rc;:S seel by eaeh issue ar-^a 
. • listeci belev;. 

u t es t: i o 1 1 s f o v I s s u o A \: e ..i T : B a c k cj r o u n d 

Wliuit is tiht: delivery system Tor education in the state? 
<i . : i um b c r of s c h o o 1 system s 

b . Voca t ional education 

(1) Adult 

( 2 ) Pes t secondary 

( 3 ) Secondary 

c. State education organizational structure 

( 1 ) State departm.ent of education 

(2) Relationship of vocational educatioi:i 
to educa t ion 

{ ]) Pol L tical relation shifts of educa tie 
to state c J o ve r nme n t and other s i cj n i - 
f 1 ca n t powe r blocks. 

What is the relative level of fmancia.l support for 
voca t ion a 1 educa t ion by source s ? 
a . I-'edera I 

b. State 

c. hocai 

What IS th:.' altitude of the state department of cnhicatio 
vocational eriucat ion toward e^valuation? 

What the afrit ade of local education aciencies tovvcird 

voc a t i o n a J. t: : 1 u c a t i o n e v a J. ua 1 1 or:i ? 

What percent of th'j ve^cat ional education students ar^; 
enroll e d i n c e o [ ; ' - r; a t i. '/ ' ' e d u c a 1 1 o n [ r o q r a m s ? 

Is the state department of vocational educ/ition viev;(}d 
as a 1 r-<:ir]e rsfi i e or roeulatory cxqcnc/? 



con tent? 



Wiia t t he - 



iitiun tor the state? 



What l^i th« er:; 
IS important ci; 



1 n t: 



• ; iL'c J r iie i n to rvi owec t ha t 
1 1 s 'i'lr-'r interview data? 



'UCisz lor.Ji i ur 



i'.-, ' f 1 ii L t ion o r Ko v 



nack l: round ' i m r ro^iL:>: i i > 



Wiial are the le'ri.'-iii .? ' :" : n i o -ms Tor the followmq 
v 'oa 1 1 ona 1 eduea t i o ' i t vi^\:^ : 



a . Studen. L 

(■ I ) 
{ ? 

\j . donp It- 1 fj i 



e at V' t.r I 



cj n a i ! 

■-a t i>:.'na J S' ■ rv i ce area ( i nc 1 udincj 



^ ) 



\'0'!:^i t. 1 (.Kia I r:c- r'/ i cc area ( includinq 

r , a :; : ; ( ; s t 

st.'C>.>n a.i ry 

\'c:-;;a t: lona 1 service area (including 



d . D r v^t-o a t 

( I i s^ ■ coiuhi r y 

{ ? ,) vi;d t I on a 1 Sij r\M ce area (includinq 
CLiurse s ) 

t ' . :<e i e t iness of employment to traminu 
' I ) S'. 'Corn k.i r y 

( .1 ) '/ocat lona.l service area ( includinq 
■ J (J u r s e s ; 
:' , >^coa:>a t ior*:a I t : t Los 

( 1 ) IVi^at do you use? 

v;ha\ de a 1 t i ■•■ :or ! ki^: ):c'y tcarms are used in collecting 
V o c a t 1 o : "1 I i '^-^ ^ : 1 1.. i a n y 1 . i c <" - m t.:^ n t data ? 

VVriat art.* rda:^ sf - u r^.;. ■ arid lakLaencos contributinq to the 
ko rma 1 1 (.):: ok d* • k :a: i k i ons ? 



a . Leu 1 1 . i t \.o:\ 

d. Kales and rc^ ; a Li t i. o:o.; 

c. State and local, ylan.s 

d. I n ko rma t .] on systvjns 



* ' . N ei t i, cj n a 1. A s s o c i n t. i o : i ( ■ t' f . 1 1 " I) i r r c I ( ; r s ^ ) l' 

Vocational Iilducat i '"vi 
L. NOIC aiKi SOIC 

5. What s tra t (H-j i'.'s ar\^ iiiuvi . -omnuiM i I'a t c> vl' ' f i a i f. i Mn:-- (>i" 

key tornis ; 

a . W r L t, t o 1 1 1 s !" u c t i ^) r i 

b . Wo r k s ho [) s c o r 1 f r o n o ^; 

c . Mo i t o r i Ti a ^a f f o r s 

(i. Othor 

C. Has the use of dofinitioas brM-r: inoa i to la 'o or ^^va L ua t. i_'d V 

a . How? 

b. When, by whom and wd tia what roiUiltV 

c . A r o r c a o r t .^-^ . j \ ' a i. 1. ..u ) 1 : c on c o ra a a 1 1 c ova 1 u \ L .1 o n 

d . Aro ova l.aat i ')a r«a-;a I ts asod tr^ iin[)rov(; tho usc^ 
: d^' : la L Li^^aa V 

7. What :aa>bb.':ns .ii"'.' aco ua ti- t-.m] iri tivj us^/ of cu/d' i a i f ioas? 



^e uo 3 t i o a s f o r Issue A r o a_ I 1 I : 
a r c*..' s s o i Co 1 I o c t i. n a PI a co mc n t f3a t a 

I . B -J c k I ; r o ■ a 1 d / ' 1 : 1 1 r o d u c t o n 

2. Please ri'-^^sca-i be the tarooesr. (.) f plannin^j and con-iac t i aq 



a . '^A ) ] ^'C t 1. VLa:: 
V) . S ci'ied u i 
c - Ao t cj r s 

Do c ^ s y ( a t' c\ u a aj y 1 1 • "i v r ' a [ a .) 1 .1 c y r f j a r d i a ( f f ) I a c e mc ? a t. 
a tud i. es ? 

a . Wh a t i- a t tie :jo 1 L cy V 
r . W r" 1 t . t- i • a V N on- w r i 1 1 c ^ n 

(.; . H r J w h <.\ s L t' e v o 1 v e d o r c: ^h a ru j o r I o r t h { a r.^ t. td i r' e e 

or tour year.s? 
d . la what form dca.^a L t c^xist and how .i s id: 
d 1 a t r L bated ? 
. Who is resr)onsible for its dc^ vc^ lo[>men t ? 
f . 1 : :a; :.o ! i ay , why aotdJ 



t. e r e ^ .. - i ! . ' ^ e I : 

uc tin^j ^ la '.x ' i'^.' ^ v Mnn\e i . d<i£ J an J, 'or utilized 

■ yar aue:: . 

IV YLS 

a , Who ' 1 ' a. 'a : ; 

\) , In whaL :ur-:a '^''V ■ a-: i a L aiai how are they 
di» tributaa 

a. How iiavo tiaa/ -/aaa.' i^.-a aa^ ovolved over the past 

thre-.' .:a- foar -.ajara . 
J. If y;ja aa\'i' i f.'^. • a a lai I aaidelines, v;hat difference 

would iL na>.<_ : -a 'lid not liave them and vice 

ve r sa ; 

t*. What aa^jo.i : ■ a.i a.it d .ua:;S exist for determiniia; who 
should Ki;'a i :^^d a a a vocational education 

i tuden L 

VVh*t sy'.''-!: ia a ■.: a..ii ' 1 laie s exist for determining 
wh«tiaa- ^a- :;'^t. i yab iv-jld by a former student is 
r<liita."i ' ■ :..a^: i a:: i V i daa 1 ' s vocational training? 

IF N<:: 

a. Why a!"'- : a* - : a r , i I ;aidL;lines not used? 

asi:^t:aac'.' ['aia;- a/ : i i a a .1 f^.^r planning and conducting 
ement at a ; i ^a: .' 

a. What .aa':aay La-a-'idea the technical assistan.aa? 

b . Wh-it: f'.jrir ao'cai teciaiical assisti^nce take? 

a. lii t:a' t*_'cia'"i i ^ai i .issistance perspective typically 

comp I. 1. aac^.' ' a T'- ^a la-im or ien ted ? 
a. What '1 1 : :\.a^aia^^ wcajLd it make if technical 

a L :i t ara:a :"d.a' adannincf and conducting placement 

a t u d 1. L ■ a V/ ( r ' . ' : ^ r f;") \' i d e d ? 



Wh'-- i. a r.a- f ■ ^-.aaaaicaJ lasistance? 



v-i a 1 u 



a ^ 



a-- )a Id i. t ...ake i f technical 
[d.anning and conducting placement 



t ud a >a W' ' ; ' • a ra,a; ided? 

la r.b'., a:ia:^^ SPA , idii; i n i a t ra t i ve role in planning and 

:a' !a'a y.'^M' ; iaC'-'aie>)t studies? 

.t . D-.'f.*:;; ::aL-' aiiA :;aa: I't -g u i remen t s , make recommenda- 
t ia J a a . )V a i.:a: ■ L y incja i to r ? 
. how aa.a wha/ iaia riu) SEA role in planning and 
rrondaatiaa local p.lacem.ent studies changed or 
V o ] V il (.) V a' r: t : a:- past three to four years? 
c . Vih a t a rio aid t: : la S MA role be and why ? 



•;07- 
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Have th'-i'L.' bot.Mi any pro:^h'r;a >.i i i' i, . a.i 1 I i ^ 

ductinn i^iaconioat stiuiu'S v;i':; r "at t ^ ^ : 



a . 


1. ti": purpose ? 






b . 


Or^orat Loa-~car ry i na i 


t out ■; 




c . 


Rosourcos (bud , 


Tseane' 1 , - I. a . ) 




cl. 


Tim Lnq 






O . 


Imp lemon tat ion r)r ut l 


. 1 i :a.it .i on / 




f . 


DeterminLnq wiio is a 


voca t i ona 1 Ovluc:a t ai 


s t udaii t 




Dotermtin i ncj re la tc^dne 


'SS o f voca t i ona 1 I ra i 


n ! na 




aiit'l i ob n ] cl b*/ fc"^rnv' 


a" s a do ri t ? 





Does your a a e n c y ^: i ^ a n y ad a n s f o r : i o v; a [ - ra- o a c !i ^ s ■ r 
chanqes for conducti a p^Lacoin^ait stadi.os as {.ai^.'y ai- 
currently conducted? If yta, , I'loasi"* oxadaLn. 

a. Vvho arid/or what brou^iat t'neso about? How will 
thijy imr^rove or chanao matters? What waS/'d s the 
S K A r r^ 1 e i n t h i s i n s t a n cc ? 

b . 1 f n o , i s then a a n y t; h 1 n cj y o u or others wo u 1 d I 1 k 
L') see chanaed in the future concernincf placement 
stud ICS? Wlia t for i ns tance ? What m i t i aei t e s 
attains t such chanae now? How would such chancrea 
i. m p r o V o p L a c mr- n t s t u d 1. e s ? 

Who ill Ida [ s aaency seems to Ij'-- primarily respons i l.d ■ • for 
the pl acement study act ivi ty? 

a. What Ls their role m the aaency? 

b . Ivhcj do tliey report to ? 

c. How do they relate to other (if any) vocational 
r: ri u r;a t i on a c i v L t: L e s 

d. Are tliere others, here or elsewhere, who i-^'lay 
{c)r have ["clayed) an i.m[:>ortant role in how 
follow~U[j is conducted and its results utili:':L'd? 

V/Iiat is the anraaal rc.asource allocation for jjlacement 
studies in your aqency? 

a . 1 s th is adequate? I f not , what i s needed? 

b . Has tiie re sou r CO al loca t ion been chanqed i n t he^ 

past throe to four ye^ars? n'ow end why? 
c . Wh o mi a k s t h e s e decisions? 

h^as the [placement study effort been moni.tored or 
V a 1 u a t e d in your a c i e n c y ? 

a . When , by whr)m and wi th what resu 1 ts? 
b . Are re por t s ava i. I ab lo conc< aai i ncj the ev, 1 1 ua t i on 
e f fort 



How : S: i : i c i: 
:as t :"iei'e b. any • r 

a . ' J f f i c e ' : i ^ : i m * 

b . U . S . 0 , , : : i a 

r . Advi sory CC'Uiui I 

J . National Assooi : 

o » Othe I" 



r .1 n : ; u t r o -r: a r a i n^: 



:ni.;rn f rouna . i n 

'■■■a what, exuen 
ra r L o t e 'i in 
'.•auca t I or: ■ 

L s a a t: o o ! 

WiaiL al.:icen:en 

;H/scr ibe n^jw 
a^ ajMcy . 

vVa. t r s u> J r J 



■ r a 
a^so: 



cerricn t Func t ior: 



• 1 :i cement i: u n c t i o n 
of vocational 



.'/:at.';.i for the placement function; 
an-.' a'/a Liable at various levels? 
ai S'..a;vi ce operates in your 

I'/a 1 h:i:j 1 (J tor operatinq placement 



r I nana i > i : 
sou r cc' o f 
St.i f r in'.: 
haaj rai n k s 



a t: ar 



support, level of sur^port. 



v; i t h o t her a c ; c n c i e s 



A r^. 'a V : At i 1 i za t ion Data 



1 . Fjackcjroana aL r.t fj' : * t i '.a, 

la p I .:i ct -mer : t a, Pa ^: : a sam i n a t O' I to significant audiences? 

a. What [^roolv.ilir^"'S aiio :'or"P.at are used to disseminate 
piacemterit da La? 

b. Wh^at ijrocajdiua.us ar^^' aa-a-a to assist the audiences to 

1 n to r a r e t a 1 aC'.:mon :;. a.i 



EKLC 



3 



- J 



■J. Hov; =ao you vic\v* ^hc SL'A ;.i>i:ry '^a a 1 aconiorit, stiUkijL 

Arc ulaceiTioat JaLa us»ja i ;na i i c i l L v '.a- axaJ.iCLtiy f^a': 

a . Personnel evaluation:"' 
1 j . r o c j r a rn e v a 1 u a t i o n 

( 1 ) p r i..^ r :i m c o n t ia i a a 1 1 : i ? 

( 2 ) proqram expans i on V 

( 3) new proqram apf^rova 1? 

( ) equi ;-'men t update? 

(j) curriculum chanqes? 

( G ) pe r so nne 1 deve 1 cqjmen t V 
'. : . y r c-a r an; nio n iter i n c; ? 



i;A'i'A _^' ^t^l<^y^^_i^^ COLLECT I <)N 
L\A1'A SnURCES 

The scju^aces of data laj documents of state and local 
agencies and personal mterviev/s with administrators and 
staff of those aqencies . The section below describes the 
types of documents thac were obtained and reviewed anu 
types of interviewees ^roni whom data were obtained. 



Doc union t s 

From the five parti cipatinq states documents were 
requested which the project staff perceived to be related 
to the issue areas. Additional documents which the state 
staff perceived to be related to one of the issue areas 
were also requested. The specific list of documents 
recjuested by the staff follows: 

1 . State* plan for vocational education 1974-75, 

I97a-7r), L97G-77, and 1977-78. 

2 . Cur r en t o rqani za t iona 1 ch dir t . 

1. Sfat(^ 1 O'j i s 1 ri t i on for or affectinci vocational 
(Education ( d t t hr^ secondary a nd p c:) s t s e c o n d a r y 
levels: for full-time and part-t ime students 
i nc -1 ud 1 n(j adu It s t ude n ts ) . 



4. Current sth^*- \^-\-*'\ j •< • i i-*y ivipidijooks and manUcA^s 

ch set iLVz':: ^: :\\z \ :: : civs and ;jrocedures, 



ijar t icula i; 1 th- r^-;. :r^-d ^^.-^ :;'lacement and 



V, Annual r'-.; : 1; t s i ■ i^..- o ' ^ :' i ; ■ - : } ;^rovi,dud to the t,S, 
v.;fLice a:^ taatM l n.':"! L074 t:hrouc:h 1977. 

:;:valuAtion ro::oi-ts s^ato vocational education 
v-roaram o:'!"orL:v:s :.i\.-:araL: by the state division 
: J f voca t Lona 1 oduca 1 1 , s t ate advi sory council 
i;or vocational oduc i^.i^'n unci other agencies, e.g. , 
state loMSlatar^j , oorurt L ss lor^.s and ad hoc groups 
; : rom 19 7 4 ry., 1 ') 7 7 . 

L::^t rumen used t^ ■ i.:';nciucL a survey of follow-up 
students a r: d t -il- . i <:: corn' : a n. y i n g instructional sheet 
md tram: :.n rr^.anaalF^ p-lus any transmittal memos 

( f rom 1 0 7 1. 'J ^ 7 '■ . 

. Important, :r.'-'::io : -.i ;n :, ; j::;: nor resoondence related to 
stude:: t p I ac<:-m';.-n. ^ t:ni l the state deemed essential 
for the nPit 1 nn,.il iP.'Mt'/^r interview team to under- 
3 1 a n d . 

'J, Orher ciofv.ir.i. -n 'n: j-..: materials related to student 
p lacemen . 

Tlve docamieri ts vn:.-r'/ cjhtain-jd ijrior to the interviews, 
v;ere reviewed, and all r'L'I^jvant data were recorded for 
project and interviL'v; purjAjsr'S. This process helped to 
provide rjro p-M:t sta:L with an understanding of the context 
and many aspects ni tho p-.l.acement data concerns that were 
addressed in thf:' i. n te r v i ev;s , 



I n t*,.'rviev;ees 

Interviev.'S w.jre c:-..j:\ducti-c\ .-/ith stato; and local staff 
Lnwp.ivjed in voctitiorial .jducation and representatives of the 
U.S. Officf.,' of iPincation, Bureau of (jccupa tional and Adult 
Pducation. Project £:-taff selected state interviewees by 
reviewmcj the state's organizational chart and selecting 
[persons who were inv< Ivod in (1) overall vocational educa- 
tion manacfement, (2) collection of placement data, 
[ I pT'-vgram s ape i: v i. s Kjn , ,.:n.\ [4) others who influence oj' 
iiave an interest in thy pl.lcem^■nt [processes, e.g., state 
adv 1 sory counci 1 , r;o(^pe ra 1 1 nc; placement agencies. Using 
these criteria, tiie state contact person was asked to review 
the p.ro]ect staff's s^jlection of .i --rerviewee s and recommend 



other ir^Lorv.iewoL'S whr.) :w\.-hL : r:j:^^".'l . v^^ .■-.] t:u.' :::..it.L' 

p.' icemen t 

Selectii'jr: : in rv l..:w^-;-s .it th- L^^^cal ucliic- 1 1 i ^ 

a _:e nc i OS wa s :j a s rni o .n t i lo s c i" i t r i - ( ] ) a ::i i. : : i mv.rr c") f 
four to z\>: persons to interview f.^-r sclioo.L, (2) a rane-.* 
o: hi^jh tD low i n vol\'emeiVc in olacemei^t ^lata, (3) .inv(jl'.'« — 
mcnt in vocatioiual ediicat i.ur:i adrni n i s t r:i t ion , quiLlanc';> , a:id 
teaching roies. 

Representatives of tiie L' . S . Officu of EducaticM^, 
Bureau of Occufja t i on.a 1 aiid Adult liciucation whf; were inti.'r- 
viewed included those cor^icerned vVitii (1) cr.'erall vocat:(3naJ 
educaticjn manacjement, (2) collection of laceiuu'n t ]"op^^^l-s 
fron thi^ states, and (J) :;roarani staff. 



!)ATA OOLLliCTlON 

Oata ccj 1 .h.'c t ion activities consisted of preparinci an 
interview manual t ■ cjuide the collection of data; traj.ninq 
interviewers to cl. ..auct interviews and record data from 
documents; Interviewmq federal, state, and local staff; 
reviewmc; ciocuments; and writinci up interviev; notes. 



mte rview Manual 



An interview manual was developed by and for use of 
project staff to assure that quality and consistent data 
were (1) collectcfi from the documents reviewed and [persons 
in terv i.ewe^d and (2) recorded in a 'Mnncr to be easily 
retrievc-d .ind analyzcKi. Thtj manual consisted of: 

1. List of c{uostioi:[S which were central to an 
i nves t icjat i(";n of each issue area, 

2. A key/code word list from the questions in each 
L s s ue area that into rvi ewe r s could use to check 
off [joints as they were covered in the intervi.e^w, 

J. Thf.' specific data that would answer each c]uestion 
and whether the data could be obtained from a 
document, an interviewee, or both. 



Co.li nq [procedures . Codii 
of codinq information on 
around ti\(- Tour sides of 
i. n ( J w i. i , ! ^. a ri e e ( 1 J. c j . T i i i s 



^q procedures c o n s i s t e d 
cards wi th holes 
the card to permit sort- 
process provided C]u i ek 



3 ' 



ERIC 



-■Cher i^.ienci f i-.-rs ^'uc:: : 

c. State 

. Local sc:\(>:A 

e , Fcde ra i 

r . I n te rv 1 cwc^'j 

; , I n te r\' j e\-;e r 

h. Document 

3. Master lisu : ^^^.^ r.".' : • ■v;t.'cs by state, 

iiit^^rview rp.a:v.:al v;as ':.set.: as the interviewers' guide 
collecting ana recorcin-: cata and was the core 
i.-r'ial used in^ :.ia.:.nin-: :.n>j;,ect staff to serve as 
r V lewe r s . 



7rainj.nj_ I n ter\^ l ew-ers 

Scufr wrio s^.^r'/t'd as i ri rviewers in the collection o\~ 
uata wer^* T'loyd McKinin.ey, Kenncy Gray, Marie Abram, Steve 

rar^.chak , and N. Mcdasi. m, 'l^raininy was provided for 

t h \j staff ] n t wo w o i* r: s n o : ) s for i n 1 1. . r v i e we r s and a pilot 
t u s t of t h e 1 n t e i; */ 1 1 j v; : j r o ct.^^ s s e s and t e c h n i q ue s in the 
n .1 d i 1 c star, a n : t w c- 1 o ^ ; a 1 education a c| e n c i e s . 



/ I'r: sno: -s 



-• 00 ] cci 1 '/t.j.-^ to-.- first staff workshop were to 

rt'View and (":r : ^- i. rt.'search assertions for the 

c<!i se s t Uf i y , 

idL^ntiiy ^nd j:.:st. ify issue? areas, 



. I T I s ( . 



r. no case study , 



4 . iden 1 1 ro/ so^.'^- 1 f ru'.'s I. ions to be addressed 
for fjci; issut, ' ur-.i , 

, cs ci:^ 1 L so c.i L' ■ ri( :, 1 r : > u' c:(jnduct 1 ncj the case s t udy , 
and 

f'. . deter:ri i.n..' detai.lc'O j-^Lans for c^.^nducting the pilot 

ca se s t ud , 



rami 1 i a r i ;:e i n r^ r'/ : -va ' r 



.n terv.1 , 



rucnn r.;'.r,.*s 



or siiTiniar : r Lroi notes dur.inu 



5, techniquos lor rocordirr: lo forrrit i o:i or irdvo< 

cards (cards r> ■ ^reiK^ od oar 1 lor with hoios around 
' r 1 ni'j t '.J r o . .t a o r o - n . l . a • a, la. . ^ ^ - 1 ^ < ^ ■ 



7 . out: 1 Lo-; :" o 'S' ■ sr udv f .i aal r^aajrl: , 

arraa':--io:it: i ard sca..:dalo for oiiot sliaLo visit 



a. ^;V.- 1: 



a bo used m contactmo and schodulinci 
!i fodorai, state, and locai aaonoies. 



P r o i o c t St a if p r o ooi r o d m a t o r i a 1 s and resources in 
advance ot^oacii workshop thus onablinq the time intorvi.owers 
soent toai-adiior t^) be i^rc^cUiCt l vo and efficient. The 
^nterv^^'v; r.anual, texts on interviews, and sur;[JOrtino 
materials -ir.. r-xanpLes u: the materials used in the v rkshoi;. 

rurtiier trair.int; v;as {provided by conductmcj a simulated 
interview' with a former director of vocational education ir^ 
a iarcje citv school. The iaiterview was conducted by one 
interviewer' With the other interviewers observinq and under 
the assum.otion that ^-he recent vocational education 
director 'was functionmq m his previous position and 
usinq real data. The planninq for the workshop, the work- 
shops themselves, and conductinq the simluated interview 
were excellent preparation for the interviewers who were 
to collect data in many settings and circumstances. The 
workshops were followed by a pilot test of the interview 
and document data collect i.on in a state division of voca 
tional educatir;n and two local education acjencies. 



EKLC 



'< iucat ion a.icnci^: 



::k: o :: oacn 



1. nclL'/: cards ana 
codes into si- 



•caaiiaues as noecicci to 
i: 1 • ;h acid 1 ovj p la cemcn t 



carri'-d out smoothly with 
: i [\ a to bo collected , documon ts 
a 1 t u \^ 1. c ws r data roc o r d i ii a- 
, jiad the analysis of data . 



' : : o r 1 1 nie s c on ve ii i e n t to 
i terv i ew schedule covered 
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ori.entation mcotinL; rcjr :jt.::u.-::L i :,:ic ::^ur:>^ns d^ji:;.' 

interviewed in the st-'ite L:i\':sion of '/cjcat iona I <..'c:ii'.Mt i>"'n 
and the i n te r v : ewe r s . ;■h^■ ee- : -n La •. i ^/n .-(.a; s i s - ' 'd f.a" i.:;^.:-.- 
duction of i^^^rsonnel, and ^r/^TView of i h-.' ccisc^ study, .md 
a review of tht: sta::e*s ■.'oca t i<jna 1 edU'::at: i c)n rie 1. ix'-.e-y 
system. llir.c to fiftoer^ stat*;* staff O'aoojns wc^ri.' int-.-r- 
vicwGG foliowinc: the aeneral orientatoaa i n to r vi( -va ; , "ditj 
individual interviews were aooat f c-rt'/- f i ve rainut^n eac:i. 



i'he interviews follov^ed ^iite : n to rview inc: technioUi:a? 
(Dexter I'^^GO* , wiier^' the i n L e r v 'Va.^ r is willine and often 
eager to iec the in ter\'iewee iden 1 1 f y ti:e :.)roh loni , the 
auestion, and ahio situation. itae basic aoijroach taro.'n. v;as 
to m form m t ^ r vie wee soft h.e data neodeci from theni-- n am.e 1 
their persu'ect i ves on placement dat^i--how the interviewea: 
understands it ao be col looted, used, and disseminated. 
An much freed jm. as ;,.oss.il~' le vaas aiven to conversa t io'i as 

:,'e r soeo t i ve s o the i n tor v i ev;ee , un ta i n ted 
r ' s im.y-i loit oommuni cat ions as often rev ■ = led 
I auestion,s about factors that are ciood or 

i : ■ n; est i t r v i e \< s , it w a s o n e i n t e r v \ e ',v e c 
: lev;^Jr . Some aspects of in terviewma vdiicl^ 
■a..' r c r a I ne d t r) foil aw v/e re : 

a'/' L n t-:.' r V ie /.a^.? s do the talkine 



ao qet th^- tr 
by m terv i ew-j 
o y a series 
not so aood. 
and one inter 
mt erv 1 ev/-.. r a 



or 'a-:amr;b>s and i I L us t r.;i L i ( ai of pomes made 



aaj to the po i nt 
; a t his own rate 



;a:^tooe<"' toirdi yu-ast ions un t i 



inl ^'r\'iew i s we 1 1 



a.' 11 '/iJO th^' 1 1 oa I adia.:- r s a r'/ te^ flush out opinion 
■ ■a ; : o o r:Ti<.i c i on 



a n 



ar'"* O'.^rce i v^al ; nowever , never 
oerc' .dvcd d L s c re oa n c i < .' a 



o r' aso Ive d iscrepanc i es when t; Iv. 

11 in to rv 1 eweu' s 



9 . take nc r - au dur lav: i n to rv i ew 



Lnt(.^rviews in LooaJ ecJucation acauioies consisted (j f 
one-to-on(.' intc^rvi^^ws for the most part with no cjroujj 



:-^o- :\s were 

a ;o:: ::y . Ir. t:erviev;s 

: C' r s o n s \< ort^ i : i u 



..r-au of '.;:::cu;-^ar 1 o::a ■ 



luin UtG 5 



7r.e snorres t 
,;c:t 1 education 



1 0 : ■ wore l n c e r v i e we : 
:nL::vios wero used 



•/ : u s : y o:j t a 1 n ea Oy 
by interviewees. 
:) o s s i b i e and utilize d 
' s lace rue n t: data . 



^ ter-en t o:\ e a-.:: . 
J i u a ' 



cnre^- ..: .1: ' : v;nd uf each aay v/ere 

: u:ej aj;i:':-:U- : ■:■ •.^.ica .i n te r\' lewe r to write a 
\ r*/ ; ' v; . '^ne t a temen t s by interview 
r.iV i i. ■-sariyri on of all the data 
'Miiioa m .jacl: i :w..- ; nni Ji' cociiny of the paragraphs 

snbnartf^ o: th- :,ar:M^in.n^ ':^ -sc r i y t ion usinq the alpha 
ma nunieric codm: .; nn-r;-.^:-; :.a-t^'.' lously described. It was 
cricicai thai ihis v;ni"^n:: r be used as scheduled and 

z'r.Jit wricma jf 1 a Le:\' • n t n \jO fully accomplishGd a 

th^:- ^uvi of v^acn in*; i:~ and sui;)sey iient: interviews 

; : .^s 1 n an 1 n - ^' r"*; : r ' n nnn:n n"y of liie conversation and 

:;"L'/:in:n: nf nh*- n-n ^n ^n-n: nnrinn nn interview. 

7':"nj v;rLttf;r. t -i tcin-j n n n v;ere subsequently typed on 
*:h..- index cariis; :ata for : ni:n:l.e code v;ere [placed on one 
cira or a sori(;s of ;.Mrnn if nrwe niian one card were 
r^niuir-a. ir: t::.: e'.'.'nr; 'n ni a cfita item related to more 
rhan one' code, a S'jn:irato c..n^-: cur series cf cards were 



/ 1 



.cd ror cn.icn a^ n; 1 1 l 'jini j coo'.^ an*-.: 
, s rmi f t.. dam n-"^ b-. rntrie\'C 



; rouped oy : nnu*.- a 
nuj f 1 ndmcis v; 1 t hi : 
.».!y terras used :n : 
: i n 1 n inn : s tn c^ n- . 



were coded separate ly . 
: d r: u i ck 1 y and to be 
la 1 wri. ters v;ould write 
s uc h as "definition of 
usinc data for all parti- 



hh KhPuRTINc; 



'.: 1 ' i ^"l n . 



w m 1 n t' "-:ra I ly 1 mkec! to the 
h- • ; a u* ' a rc:as . 



he a j ^ p^ r o a c h taken 




3' '. . 



in the analysis of the data emphasizGd a comparative 
treatment of the issue areas. Attention was focused on 
across-state analysis of the -actors that were identified 
as potentially related to the differonces in the reportec! 
job placement rates. The output of this activity was a 
comparative analysis or synthesis of each of the rosoarch 
issue area questions across the five states and thu' federal 
level . 

Where appropriate, the data collected in the stuciy were 
placed in framework to provide crraphic depiction. Compari- 
sons wore . .own graphically across hicjh, middle, and low 
placement s ta tes . Background and key f indincjs a ro repo rteci 
in the body of this document. All data analysis arc repoi'tcd 
in this report. 
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